2023-07-14 09:06:26,025 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-14 09:06:26,260 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408 2023-07-14 09:06:26,274 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-14 09:06:26,275 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/cluster_708bca9d-6cdf-4da8-0283-1300549de5a3, deleteOnExit=true 2023-07-14 09:06:26,275 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-14 09:06:26,275 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/test.cache.data in system properties and HBase conf 2023-07-14 09:06:26,276 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/hadoop.tmp.dir in system properties and HBase conf 2023-07-14 09:06:26,276 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/hadoop.log.dir in system properties and HBase conf 2023-07-14 09:06:26,277 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-14 09:06:26,277 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-14 09:06:26,277 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-14 09:06:26,391 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-14 09:06:26,772 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-14 09:06:26,776 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-14 09:06:26,777 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-14 09:06:26,777 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-14 09:06:26,777 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-14 09:06:26,777 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-14 09:06:26,778 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-14 09:06:26,778 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-14 09:06:26,778 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-14 09:06:26,779 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-14 09:06:26,779 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/nfs.dump.dir in system properties and HBase conf 2023-07-14 09:06:26,780 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/java.io.tmpdir in system properties and HBase conf 2023-07-14 09:06:26,780 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-14 09:06:26,780 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-14 09:06:26,781 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-14 09:06:27,277 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-14 09:06:27,281 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-14 09:06:27,557 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-14 09:06:27,706 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-14 09:06:27,720 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-14 09:06:27,753 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-14 09:06:27,784 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/java.io.tmpdir/Jetty_localhost_42733_hdfs____phrqyr/webapp 2023-07-14 09:06:27,935 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42733 2023-07-14 09:06:27,945 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-14 09:06:27,945 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-14 09:06:28,334 WARN [Listener at localhost/46473] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-14 09:06:28,432 WARN [Listener at localhost/46473] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-14 09:06:28,449 WARN [Listener at localhost/46473] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-14 09:06:28,454 INFO [Listener at localhost/46473] log.Slf4jLog(67): jetty-6.1.26 2023-07-14 09:06:28,458 INFO [Listener at localhost/46473] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/java.io.tmpdir/Jetty_localhost_40045_datanode____qgqa70/webapp 2023-07-14 09:06:28,553 INFO [Listener at localhost/46473] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40045 2023-07-14 09:06:28,839 WARN [Listener at localhost/34569] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-14 09:06:28,851 WARN [Listener at localhost/34569] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-14 09:06:28,854 WARN [Listener at localhost/34569] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-14 09:06:28,857 INFO [Listener at localhost/34569] log.Slf4jLog(67): jetty-6.1.26 2023-07-14 09:06:28,863 INFO [Listener at localhost/34569] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/java.io.tmpdir/Jetty_localhost_33723_datanode____.xdy1tt/webapp 2023-07-14 09:06:28,967 INFO [Listener at localhost/34569] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33723 2023-07-14 09:06:28,976 WARN [Listener at localhost/45201] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-14 09:06:29,382 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x9dcbff08f5fad7bd: Processing first storage report for DS-46c52e53-4cf7-418f-9976-a339137f48a3 from datanode 250924d0-68c2-457f-b965-8dbeecbad572 2023-07-14 09:06:29,384 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x9dcbff08f5fad7bd: from storage DS-46c52e53-4cf7-418f-9976-a339137f48a3 node DatanodeRegistration(127.0.0.1:33565, datanodeUuid=250924d0-68c2-457f-b965-8dbeecbad572, infoPort=41453, infoSecurePort=0, ipcPort=34569, storageInfo=lv=-57;cid=testClusterID;nsid=1827630822;c=1689325587359), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-14 09:06:29,384 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2563e40aa1c5032b: Processing first storage report for DS-06b9980b-5da4-4372-b119-090a73b88c97 from datanode 061ef11e-14fd-4f2e-801d-3d083e37c651 2023-07-14 09:06:29,384 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2563e40aa1c5032b: from storage DS-06b9980b-5da4-4372-b119-090a73b88c97 node DatanodeRegistration(127.0.0.1:41183, datanodeUuid=061ef11e-14fd-4f2e-801d-3d083e37c651, infoPort=37617, infoSecurePort=0, ipcPort=45201, storageInfo=lv=-57;cid=testClusterID;nsid=1827630822;c=1689325587359), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-14 09:06:29,384 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x9dcbff08f5fad7bd: Processing first storage report for DS-718da854-42b9-4d93-a227-54337aabb196 from datanode 250924d0-68c2-457f-b965-8dbeecbad572 2023-07-14 09:06:29,384 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x9dcbff08f5fad7bd: from storage DS-718da854-42b9-4d93-a227-54337aabb196 node DatanodeRegistration(127.0.0.1:33565, datanodeUuid=250924d0-68c2-457f-b965-8dbeecbad572, infoPort=41453, infoSecurePort=0, ipcPort=34569, storageInfo=lv=-57;cid=testClusterID;nsid=1827630822;c=1689325587359), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-14 09:06:29,384 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2563e40aa1c5032b: Processing first storage report for DS-ffbe975d-4a04-4166-84e2-14b388b6a037 from datanode 061ef11e-14fd-4f2e-801d-3d083e37c651 2023-07-14 09:06:29,385 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2563e40aa1c5032b: from storage DS-ffbe975d-4a04-4166-84e2-14b388b6a037 node DatanodeRegistration(127.0.0.1:41183, datanodeUuid=061ef11e-14fd-4f2e-801d-3d083e37c651, infoPort=37617, infoSecurePort=0, ipcPort=45201, storageInfo=lv=-57;cid=testClusterID;nsid=1827630822;c=1689325587359), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-14 09:06:29,415 DEBUG [Listener at localhost/45201] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408 2023-07-14 09:06:29,473 INFO [Listener at localhost/45201] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/cluster_708bca9d-6cdf-4da8-0283-1300549de5a3/zookeeper_0, clientPort=52519, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/cluster_708bca9d-6cdf-4da8-0283-1300549de5a3/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/cluster_708bca9d-6cdf-4da8-0283-1300549de5a3/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-14 09:06:29,485 INFO [Listener at localhost/45201] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=52519 2023-07-14 09:06:29,493 INFO [Listener at localhost/45201] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 09:06:29,494 INFO [Listener at localhost/45201] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 09:06:30,141 INFO [Listener at localhost/45201] util.FSUtils(471): Created version file at hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4 with version=8 2023-07-14 09:06:30,142 INFO [Listener at localhost/45201] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/hbase-staging 2023-07-14 09:06:30,438 INFO [Listener at localhost/45201] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-14 09:06:30,892 INFO [Listener at localhost/45201] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-07-14 09:06:30,923 INFO [Listener at localhost/45201] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 09:06:30,924 INFO [Listener at localhost/45201] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-14 09:06:30,924 INFO [Listener at localhost/45201] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-14 09:06:30,924 INFO [Listener at localhost/45201] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 09:06:30,924 INFO [Listener at localhost/45201] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-14 09:06:31,061 INFO [Listener at localhost/45201] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-14 09:06:31,131 DEBUG [Listener at localhost/45201] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-14 09:06:31,223 INFO [Listener at localhost/45201] ipc.NettyRpcServer(120): Bind to /172.31.2.10:44817 2023-07-14 09:06:31,233 INFO [Listener at localhost/45201] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 09:06:31,235 INFO [Listener at localhost/45201] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 09:06:31,254 INFO [Listener at localhost/45201] zookeeper.RecoverableZooKeeper(93): Process identifier=master:44817 connecting to ZooKeeper ensemble=127.0.0.1:52519 2023-07-14 09:06:31,292 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:448170x0, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-14 09:06:31,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:44817-0x101630797730000 connected 2023-07-14 09:06:31,321 DEBUG [Listener at localhost/45201] zookeeper.ZKUtil(164): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-14 09:06:31,322 DEBUG [Listener at localhost/45201] zookeeper.ZKUtil(164): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 09:06:31,325 DEBUG [Listener at localhost/45201] zookeeper.ZKUtil(164): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-14 09:06:31,334 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44817 2023-07-14 09:06:31,334 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44817 2023-07-14 09:06:31,334 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44817 2023-07-14 09:06:31,335 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44817 2023-07-14 09:06:31,335 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44817 2023-07-14 09:06:31,341 INFO [Listener at localhost/45201] master.HMaster(444): hbase.rootdir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4, hbase.cluster.distributed=false 2023-07-14 09:06:31,412 INFO [Listener at localhost/45201] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-07-14 09:06:31,412 INFO [Listener at localhost/45201] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 09:06:31,412 INFO [Listener at localhost/45201] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-14 09:06:31,413 INFO [Listener at localhost/45201] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-14 09:06:31,413 INFO [Listener at localhost/45201] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 09:06:31,413 INFO [Listener at localhost/45201] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-14 09:06:31,418 INFO [Listener at localhost/45201] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-14 09:06:31,422 INFO [Listener at localhost/45201] ipc.NettyRpcServer(120): Bind to /172.31.2.10:44241 2023-07-14 09:06:31,424 INFO [Listener at localhost/45201] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-14 09:06:31,430 DEBUG [Listener at localhost/45201] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-14 09:06:31,431 INFO [Listener at localhost/45201] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 09:06:31,434 INFO [Listener at localhost/45201] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 09:06:31,435 INFO [Listener at localhost/45201] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:44241 connecting to ZooKeeper ensemble=127.0.0.1:52519 2023-07-14 09:06:31,440 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:442410x0, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-14 09:06:31,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:44241-0x101630797730001 connected 2023-07-14 09:06:31,441 DEBUG [Listener at localhost/45201] zookeeper.ZKUtil(164): regionserver:44241-0x101630797730001, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-14 09:06:31,443 DEBUG [Listener at localhost/45201] zookeeper.ZKUtil(164): regionserver:44241-0x101630797730001, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 09:06:31,444 DEBUG [Listener at localhost/45201] zookeeper.ZKUtil(164): regionserver:44241-0x101630797730001, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-14 09:06:31,444 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44241 2023-07-14 09:06:31,444 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44241 2023-07-14 09:06:31,445 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44241 2023-07-14 09:06:31,445 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44241 2023-07-14 09:06:31,446 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44241 2023-07-14 09:06:31,460 INFO [Listener at localhost/45201] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-07-14 09:06:31,460 INFO [Listener at localhost/45201] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 09:06:31,460 INFO [Listener at localhost/45201] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-14 09:06:31,461 INFO [Listener at localhost/45201] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-14 09:06:31,461 INFO [Listener at localhost/45201] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 09:06:31,461 INFO [Listener at localhost/45201] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-14 09:06:31,461 INFO [Listener at localhost/45201] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-14 09:06:31,462 INFO [Listener at localhost/45201] ipc.NettyRpcServer(120): Bind to /172.31.2.10:33419 2023-07-14 09:06:31,463 INFO [Listener at localhost/45201] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-14 09:06:31,464 DEBUG [Listener at localhost/45201] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-14 09:06:31,465 INFO [Listener at localhost/45201] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 09:06:31,467 INFO [Listener at localhost/45201] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 09:06:31,469 INFO [Listener at localhost/45201] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33419 connecting to ZooKeeper ensemble=127.0.0.1:52519 2023-07-14 09:06:31,473 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:334190x0, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-14 09:06:31,474 DEBUG [Listener at localhost/45201] zookeeper.ZKUtil(164): regionserver:334190x0, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-14 09:06:31,475 DEBUG [Listener at localhost/45201] zookeeper.ZKUtil(164): regionserver:334190x0, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 09:06:31,475 DEBUG [Listener at localhost/45201] zookeeper.ZKUtil(164): regionserver:334190x0, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-14 09:06:31,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:33419-0x101630797730002 connected 2023-07-14 09:06:31,478 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33419 2023-07-14 09:06:31,478 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33419 2023-07-14 09:06:31,478 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33419 2023-07-14 09:06:31,479 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33419 2023-07-14 09:06:31,480 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33419 2023-07-14 09:06:31,481 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,44817,1689325590279 2023-07-14 09:06:31,490 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-14 09:06:31,491 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,44817,1689325590279 2023-07-14 09:06:31,515 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:44241-0x101630797730001, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-14 09:06:31,515 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-14 09:06:31,515 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:33419-0x101630797730002, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-14 09:06:31,516 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 09:06:31,517 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-14 09:06:31,518 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,44817,1689325590279 from backup master directory 2023-07-14 09:06:31,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-14 09:06:31,523 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,44817,1689325590279 2023-07-14 09:06:31,523 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-14 09:06:31,524 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-14 09:06:31,524 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase9.apache.org,44817,1689325590279 2023-07-14 09:06:31,527 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-14 09:06:31,529 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-14 09:06:31,615 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/hbase.id with ID: 770ab0a7-e7c2-47a2-ae74-1d796eeb3b23 2023-07-14 09:06:31,658 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 09:06:31,673 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 09:06:31,717 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x52be7963 to 127.0.0.1:52519 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 09:06:31,749 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b9c6a4b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 09:06:31,771 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-14 09:06:31,773 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-14 09:06:31,792 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-14 09:06:31,792 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-14 09:06:31,795 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-14 09:06:31,799 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-14 09:06:31,800 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 09:06:31,832 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/MasterData/data/master/store-tmp 2023-07-14 09:06:31,863 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 09:06:31,864 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-14 09:06:31,864 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 09:06:31,864 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 09:06:31,864 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-14 09:06:31,864 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 09:06:31,864 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 09:06:31,864 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-14 09:06:31,866 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/MasterData/WALs/jenkins-hbase9.apache.org,44817,1689325590279 2023-07-14 09:06:31,888 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C44817%2C1689325590279, suffix=, logDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/MasterData/WALs/jenkins-hbase9.apache.org,44817,1689325590279, archiveDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/MasterData/oldWALs, maxLogs=10 2023-07-14 09:06:31,942 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33565,DS-46c52e53-4cf7-418f-9976-a339137f48a3,DISK] 2023-07-14 09:06:31,942 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41183,DS-06b9980b-5da4-4372-b119-090a73b88c97,DISK] 2023-07-14 09:06:31,949 DEBUG [RS-EventLoopGroup-4-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 09:06:32,011 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/MasterData/WALs/jenkins-hbase9.apache.org,44817,1689325590279/jenkins-hbase9.apache.org%2C44817%2C1689325590279.1689325591898 2023-07-14 09:06:32,011 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41183,DS-06b9980b-5da4-4372-b119-090a73b88c97,DISK], DatanodeInfoWithStorage[127.0.0.1:33565,DS-46c52e53-4cf7-418f-9976-a339137f48a3,DISK]] 2023-07-14 09:06:32,012 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-14 09:06:32,012 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 09:06:32,015 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-14 09:06:32,016 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-14 09:06:32,071 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-14 09:06:32,078 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-14 09:06:32,101 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-14 09:06:32,113 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:32,122 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-14 09:06:32,124 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-14 09:06:32,145 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-14 09:06:32,151 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-14 09:06:32,152 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75064312, jitterRate=0.11854541301727295}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 09:06:32,153 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-14 09:06:32,158 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-14 09:06:32,178 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-14 09:06:32,178 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-14 09:06:32,181 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-14 09:06:32,183 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-14 09:06:32,214 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-07-14 09:06:32,214 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-14 09:06:32,242 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-14 09:06:32,247 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-14 09:06:32,271 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-14 09:06:32,275 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-14 09:06:32,276 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-14 09:06:32,281 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-14 09:06:32,285 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-14 09:06:32,287 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 09:06:32,289 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-14 09:06:32,289 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-14 09:06:32,300 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-14 09:06:32,304 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-14 09:06:32,304 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:44241-0x101630797730001, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-14 09:06:32,304 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:33419-0x101630797730002, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-14 09:06:32,304 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 09:06:32,305 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase9.apache.org,44817,1689325590279, sessionid=0x101630797730000, setting cluster-up flag (Was=false) 2023-07-14 09:06:32,324 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 09:06:32,330 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-14 09:06:32,331 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,44817,1689325590279 2023-07-14 09:06:32,337 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 09:06:32,343 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-14 09:06:32,344 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,44817,1689325590279 2023-07-14 09:06:32,347 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/.hbase-snapshot/.tmp 2023-07-14 09:06:32,383 INFO [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(951): ClusterId : 770ab0a7-e7c2-47a2-ae74-1d796eeb3b23 2023-07-14 09:06:32,383 INFO [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer(951): ClusterId : 770ab0a7-e7c2-47a2-ae74-1d796eeb3b23 2023-07-14 09:06:32,388 DEBUG [RS:0;jenkins-hbase9:44241] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-14 09:06:32,388 DEBUG [RS:1;jenkins-hbase9:33419] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-14 09:06:32,394 DEBUG [RS:1;jenkins-hbase9:33419] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-14 09:06:32,394 DEBUG [RS:0;jenkins-hbase9:44241] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-14 09:06:32,394 DEBUG [RS:1;jenkins-hbase9:33419] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-14 09:06:32,394 DEBUG [RS:0;jenkins-hbase9:44241] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-14 09:06:32,398 DEBUG [RS:1;jenkins-hbase9:33419] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-14 09:06:32,398 DEBUG [RS:0;jenkins-hbase9:44241] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-14 09:06:32,399 DEBUG [RS:1;jenkins-hbase9:33419] zookeeper.ReadOnlyZKClient(139): Connect 0x62aad9af to 127.0.0.1:52519 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 09:06:32,400 DEBUG [RS:0;jenkins-hbase9:44241] zookeeper.ReadOnlyZKClient(139): Connect 0x55da604d to 127.0.0.1:52519 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 09:06:32,407 DEBUG [RS:1;jenkins-hbase9:33419] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c6d3b8a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 09:06:32,407 DEBUG [RS:0;jenkins-hbase9:44241] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b1d9486, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 09:06:32,407 DEBUG [RS:1;jenkins-hbase9:33419] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6852b2df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-14 09:06:32,407 DEBUG [RS:0;jenkins-hbase9:44241] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f220b3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-14 09:06:32,430 DEBUG [RS:0;jenkins-hbase9:44241] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase9:44241 2023-07-14 09:06:32,433 INFO [RS:0;jenkins-hbase9:44241] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-14 09:06:32,433 INFO [RS:0;jenkins-hbase9:44241] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-14 09:06:32,433 DEBUG [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer(1022): About to register with Master. 2023-07-14 09:06:32,434 DEBUG [RS:1;jenkins-hbase9:33419] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase9:33419 2023-07-14 09:06:32,435 INFO [RS:1;jenkins-hbase9:33419] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-14 09:06:32,435 INFO [RS:1;jenkins-hbase9:33419] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-14 09:06:32,435 DEBUG [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(1022): About to register with Master. 2023-07-14 09:06:32,436 INFO [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase9.apache.org,44817,1689325590279 with isa=jenkins-hbase9.apache.org/172.31.2.10:44241, startcode=1689325591411 2023-07-14 09:06:32,436 INFO [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase9.apache.org,44817,1689325590279 with isa=jenkins-hbase9.apache.org/172.31.2.10:33419, startcode=1689325591459 2023-07-14 09:06:32,453 DEBUG [RS:1;jenkins-hbase9:33419] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-14 09:06:32,453 DEBUG [RS:0;jenkins-hbase9:44241] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-14 09:06:32,466 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-14 09:06:32,477 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-14 09:06:32,477 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-14 09:06:32,477 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-14 09:06:32,477 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-14 09:06:32,477 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-07-14 09:06:32,477 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,478 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-14 09:06:32,478 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,482 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689325622482 2023-07-14 09:06:32,484 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-14 09:06:32,496 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-14 09:06:32,496 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-14 09:06:32,496 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-14 09:06:32,502 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-14 09:06:32,503 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-14 09:06:32,504 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-14 09:06:32,505 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-14 09:06:32,506 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-14 09:06:32,506 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:32,510 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-14 09:06:32,512 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-14 09:06:32,512 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-14 09:06:32,515 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-14 09:06:32,515 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-14 09:06:32,515 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52515, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-14 09:06:32,515 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:36183, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-14 09:06:32,517 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689325592517,5,FailOnTimeoutGroup] 2023-07-14 09:06:32,518 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689325592517,5,FailOnTimeoutGroup] 2023-07-14 09:06:32,518 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:32,518 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-14 09:06:32,520 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:32,520 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:32,529 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44817] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:32,544 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44817] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,44241,1689325591411 2023-07-14 09:06:32,545 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-14 09:06:32,546 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-14 09:06:32,546 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4 2023-07-14 09:06:32,550 DEBUG [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4 2023-07-14 09:06:32,550 DEBUG [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4 2023-07-14 09:06:32,550 DEBUG [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:46473 2023-07-14 09:06:32,550 DEBUG [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:46473 2023-07-14 09:06:32,550 DEBUG [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-14 09:06:32,550 DEBUG [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-14 09:06:32,561 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 09:06:32,563 DEBUG [RS:1;jenkins-hbase9:33419] zookeeper.ZKUtil(162): regionserver:33419-0x101630797730002, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:32,563 DEBUG [RS:0;jenkins-hbase9:44241] zookeeper.ZKUtil(162): regionserver:44241-0x101630797730001, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,44241,1689325591411 2023-07-14 09:06:32,563 WARN [RS:1;jenkins-hbase9:33419] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-14 09:06:32,563 WARN [RS:0;jenkins-hbase9:44241] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-14 09:06:32,564 INFO [RS:1;jenkins-hbase9:33419] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 09:06:32,564 INFO [RS:0;jenkins-hbase9:44241] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 09:06:32,565 DEBUG [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(1948): logDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:32,565 DEBUG [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer(1948): logDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411 2023-07-14 09:06:32,566 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,33419,1689325591459] 2023-07-14 09:06:32,566 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,44241,1689325591411] 2023-07-14 09:06:32,575 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 09:06:32,578 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-14 09:06:32,580 DEBUG [RS:0;jenkins-hbase9:44241] zookeeper.ZKUtil(162): regionserver:44241-0x101630797730001, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:32,580 DEBUG [RS:1;jenkins-hbase9:33419] zookeeper.ZKUtil(162): regionserver:33419-0x101630797730002, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:32,581 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/info 2023-07-14 09:06:32,581 DEBUG [RS:0;jenkins-hbase9:44241] zookeeper.ZKUtil(162): regionserver:44241-0x101630797730001, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,44241,1689325591411 2023-07-14 09:06:32,581 DEBUG [RS:1;jenkins-hbase9:33419] zookeeper.ZKUtil(162): regionserver:33419-0x101630797730002, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,44241,1689325591411 2023-07-14 09:06:32,582 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-14 09:06:32,583 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:32,583 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-14 09:06:32,586 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/rep_barrier 2023-07-14 09:06:32,586 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-14 09:06:32,587 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:32,588 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-14 09:06:32,590 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/table 2023-07-14 09:06:32,590 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-14 09:06:32,591 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:32,593 DEBUG [RS:1;jenkins-hbase9:33419] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-14 09:06:32,593 DEBUG [RS:0;jenkins-hbase9:44241] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-14 09:06:32,594 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740 2023-07-14 09:06:32,595 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740 2023-07-14 09:06:32,599 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-14 09:06:32,602 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-14 09:06:32,604 INFO [RS:1;jenkins-hbase9:33419] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-14 09:06:32,604 INFO [RS:0;jenkins-hbase9:44241] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-14 09:06:32,606 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-14 09:06:32,607 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67020498, jitterRate=-0.001316756010055542}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-14 09:06:32,607 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-14 09:06:32,607 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-14 09:06:32,608 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-14 09:06:32,608 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-14 09:06:32,608 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-14 09:06:32,608 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-14 09:06:32,609 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-14 09:06:32,609 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-14 09:06:32,615 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-14 09:06:32,615 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-14 09:06:32,624 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-14 09:06:32,629 INFO [RS:0;jenkins-hbase9:44241] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-14 09:06:32,629 INFO [RS:1;jenkins-hbase9:33419] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-14 09:06:32,632 INFO [RS:0;jenkins-hbase9:44241] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-14 09:06:32,632 INFO [RS:1;jenkins-hbase9:33419] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-14 09:06:32,633 INFO [RS:0;jenkins-hbase9:44241] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:32,633 INFO [RS:1;jenkins-hbase9:33419] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:32,634 INFO [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-14 09:06:32,634 INFO [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-14 09:06:32,639 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-14 09:06:32,641 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-14 09:06:32,643 INFO [RS:1;jenkins-hbase9:33419] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:32,643 INFO [RS:0;jenkins-hbase9:44241] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:32,643 DEBUG [RS:1;jenkins-hbase9:33419] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,643 DEBUG [RS:0;jenkins-hbase9:44241] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,644 DEBUG [RS:1;jenkins-hbase9:33419] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,644 DEBUG [RS:0;jenkins-hbase9:44241] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,644 DEBUG [RS:1;jenkins-hbase9:33419] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,644 DEBUG [RS:0;jenkins-hbase9:44241] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,644 DEBUG [RS:1;jenkins-hbase9:33419] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,644 DEBUG [RS:0;jenkins-hbase9:44241] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,644 DEBUG [RS:1;jenkins-hbase9:33419] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,644 DEBUG [RS:0;jenkins-hbase9:44241] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,644 DEBUG [RS:1;jenkins-hbase9:33419] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-14 09:06:32,645 DEBUG [RS:0;jenkins-hbase9:44241] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-14 09:06:32,645 DEBUG [RS:1;jenkins-hbase9:33419] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,645 DEBUG [RS:0;jenkins-hbase9:44241] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,645 DEBUG [RS:1;jenkins-hbase9:33419] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,645 DEBUG [RS:0;jenkins-hbase9:44241] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,645 DEBUG [RS:1;jenkins-hbase9:33419] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,645 DEBUG [RS:0;jenkins-hbase9:44241] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,645 DEBUG [RS:1;jenkins-hbase9:33419] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,645 DEBUG [RS:0;jenkins-hbase9:44241] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:32,646 INFO [RS:1;jenkins-hbase9:33419] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:32,646 INFO [RS:1;jenkins-hbase9:33419] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:32,647 INFO [RS:1;jenkins-hbase9:33419] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:32,647 INFO [RS:0;jenkins-hbase9:44241] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:32,647 INFO [RS:0;jenkins-hbase9:44241] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:32,647 INFO [RS:0;jenkins-hbase9:44241] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:32,663 INFO [RS:0;jenkins-hbase9:44241] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-14 09:06:32,663 INFO [RS:1;jenkins-hbase9:33419] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-14 09:06:32,666 INFO [RS:1;jenkins-hbase9:33419] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,33419,1689325591459-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:32,666 INFO [RS:0;jenkins-hbase9:44241] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,44241,1689325591411-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:32,688 INFO [RS:0;jenkins-hbase9:44241] regionserver.Replication(203): jenkins-hbase9.apache.org,44241,1689325591411 started 2023-07-14 09:06:32,688 INFO [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer(1637): Serving as jenkins-hbase9.apache.org,44241,1689325591411, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:44241, sessionid=0x101630797730001 2023-07-14 09:06:32,688 INFO [RS:1;jenkins-hbase9:33419] regionserver.Replication(203): jenkins-hbase9.apache.org,33419,1689325591459 started 2023-07-14 09:06:32,689 INFO [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(1637): Serving as jenkins-hbase9.apache.org,33419,1689325591459, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:33419, sessionid=0x101630797730002 2023-07-14 09:06:32,689 DEBUG [RS:1;jenkins-hbase9:33419] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-14 09:06:32,689 DEBUG [RS:0;jenkins-hbase9:44241] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-14 09:06:32,689 DEBUG [RS:1;jenkins-hbase9:33419] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:32,689 DEBUG [RS:0;jenkins-hbase9:44241] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,44241,1689325591411 2023-07-14 09:06:32,689 DEBUG [RS:1;jenkins-hbase9:33419] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33419,1689325591459' 2023-07-14 09:06:32,689 DEBUG [RS:0;jenkins-hbase9:44241] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,44241,1689325591411' 2023-07-14 09:06:32,690 DEBUG [RS:0;jenkins-hbase9:44241] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-14 09:06:32,690 DEBUG [RS:1;jenkins-hbase9:33419] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-14 09:06:32,690 DEBUG [RS:0;jenkins-hbase9:44241] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-14 09:06:32,690 DEBUG [RS:1;jenkins-hbase9:33419] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-14 09:06:32,691 DEBUG [RS:0;jenkins-hbase9:44241] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-14 09:06:32,691 DEBUG [RS:1;jenkins-hbase9:33419] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-14 09:06:32,691 DEBUG [RS:0;jenkins-hbase9:44241] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-14 09:06:32,691 DEBUG [RS:1;jenkins-hbase9:33419] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-14 09:06:32,691 DEBUG [RS:0;jenkins-hbase9:44241] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,44241,1689325591411 2023-07-14 09:06:32,691 DEBUG [RS:1;jenkins-hbase9:33419] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:32,692 DEBUG [RS:1;jenkins-hbase9:33419] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33419,1689325591459' 2023-07-14 09:06:32,692 DEBUG [RS:0;jenkins-hbase9:44241] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,44241,1689325591411' 2023-07-14 09:06:32,692 DEBUG [RS:0;jenkins-hbase9:44241] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-14 09:06:32,692 DEBUG [RS:1;jenkins-hbase9:33419] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-14 09:06:32,692 DEBUG [RS:0;jenkins-hbase9:44241] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-14 09:06:32,692 DEBUG [RS:1;jenkins-hbase9:33419] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-14 09:06:32,693 DEBUG [RS:1;jenkins-hbase9:33419] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-14 09:06:32,693 DEBUG [RS:0;jenkins-hbase9:44241] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-14 09:06:32,693 INFO [RS:1;jenkins-hbase9:33419] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-14 09:06:32,693 INFO [RS:0;jenkins-hbase9:44241] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-14 09:06:32,693 INFO [RS:1;jenkins-hbase9:33419] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-14 09:06:32,694 INFO [RS:0;jenkins-hbase9:44241] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-14 09:06:32,793 DEBUG [jenkins-hbase9:44817] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-14 09:06:32,797 DEBUG [jenkins-hbase9:44817] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-07-14 09:06:32,802 DEBUG [jenkins-hbase9:44817] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 09:06:32,802 DEBUG [jenkins-hbase9:44817] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-14 09:06:32,802 DEBUG [jenkins-hbase9:44817] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-14 09:06:32,805 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,44241,1689325591411, state=OPENING 2023-07-14 09:06:32,805 INFO [RS:1;jenkins-hbase9:33419] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C33419%2C1689325591459, suffix=, logDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459, archiveDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/oldWALs, maxLogs=32 2023-07-14 09:06:32,805 INFO [RS:0;jenkins-hbase9:44241] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C44241%2C1689325591411, suffix=, logDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411, archiveDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/oldWALs, maxLogs=32 2023-07-14 09:06:32,814 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-14 09:06:32,816 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 09:06:32,816 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 09:06:32,820 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,44241,1689325591411}] 2023-07-14 09:06:32,829 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41183,DS-06b9980b-5da4-4372-b119-090a73b88c97,DISK] 2023-07-14 09:06:32,829 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33565,DS-46c52e53-4cf7-418f-9976-a339137f48a3,DISK] 2023-07-14 09:06:32,834 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33565,DS-46c52e53-4cf7-418f-9976-a339137f48a3,DISK] 2023-07-14 09:06:32,834 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41183,DS-06b9980b-5da4-4372-b119-090a73b88c97,DISK] 2023-07-14 09:06:32,840 INFO [RS:1;jenkins-hbase9:33419] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459/jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810 2023-07-14 09:06:32,843 DEBUG [RS:1;jenkins-hbase9:33419] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33565,DS-46c52e53-4cf7-418f-9976-a339137f48a3,DISK], DatanodeInfoWithStorage[127.0.0.1:41183,DS-06b9980b-5da4-4372-b119-090a73b88c97,DISK]] 2023-07-14 09:06:32,844 INFO [RS:0;jenkins-hbase9:44241] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411/jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810 2023-07-14 09:06:32,844 DEBUG [RS:0;jenkins-hbase9:44241] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41183,DS-06b9980b-5da4-4372-b119-090a73b88c97,DISK], DatanodeInfoWithStorage[127.0.0.1:33565,DS-46c52e53-4cf7-418f-9976-a339137f48a3,DISK]] 2023-07-14 09:06:33,012 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,44241,1689325591411 2023-07-14 09:06:33,014 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-14 09:06:33,017 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:51304, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-14 09:06:33,028 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-14 09:06:33,028 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 09:06:33,031 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta, suffix=.meta, logDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411, archiveDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/oldWALs, maxLogs=32 2023-07-14 09:06:33,050 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41183,DS-06b9980b-5da4-4372-b119-090a73b88c97,DISK] 2023-07-14 09:06:33,051 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33565,DS-46c52e53-4cf7-418f-9976-a339137f48a3,DISK] 2023-07-14 09:06:33,058 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411/jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta 2023-07-14 09:06:33,059 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41183,DS-06b9980b-5da4-4372-b119-090a73b88c97,DISK], DatanodeInfoWithStorage[127.0.0.1:33565,DS-46c52e53-4cf7-418f-9976-a339137f48a3,DISK]] 2023-07-14 09:06:33,059 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-14 09:06:33,061 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-14 09:06:33,078 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-14 09:06:33,117 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-14 09:06:33,122 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-14 09:06:33,123 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 09:06:33,123 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-14 09:06:33,123 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-14 09:06:33,126 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-14 09:06:33,128 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/info 2023-07-14 09:06:33,128 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/info 2023-07-14 09:06:33,129 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-14 09:06:33,129 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:33,130 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-14 09:06:33,131 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/rep_barrier 2023-07-14 09:06:33,131 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/rep_barrier 2023-07-14 09:06:33,132 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-14 09:06:33,132 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:33,133 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-14 09:06:33,134 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/table 2023-07-14 09:06:33,134 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/table 2023-07-14 09:06:33,134 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-14 09:06:33,135 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:33,137 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740 2023-07-14 09:06:33,139 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740 2023-07-14 09:06:33,143 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-14 09:06:33,145 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-14 09:06:33,146 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60197792, jitterRate=-0.10298299789428711}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-14 09:06:33,146 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-14 09:06:33,155 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689325593005 2023-07-14 09:06:33,172 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-14 09:06:33,173 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-14 09:06:33,173 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,44241,1689325591411, state=OPEN 2023-07-14 09:06:33,176 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-14 09:06:33,176 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 09:06:33,180 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-14 09:06:33,180 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,44241,1689325591411 in 357 msec 2023-07-14 09:06:33,185 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-14 09:06:33,185 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 557 msec 2023-07-14 09:06:33,190 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 799 msec 2023-07-14 09:06:33,190 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689325593190, completionTime=-1 2023-07-14 09:06:33,191 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-14 09:06:33,191 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-14 09:06:33,258 DEBUG [hconnection-0x3defed0f-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 09:06:33,261 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:51308, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 09:06:33,278 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-14 09:06:33,278 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689325653278 2023-07-14 09:06:33,278 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689325713278 2023-07-14 09:06:33,278 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 87 msec 2023-07-14 09:06:33,301 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,44817,1689325590279-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:33,301 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,44817,1689325590279-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:33,301 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,44817,1689325590279-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:33,303 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:44817, period=300000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:33,303 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:33,308 DEBUG [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-14 09:06:33,316 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-14 09:06:33,318 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-14 09:06:33,327 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-14 09:06:33,329 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-14 09:06:33,331 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-14 09:06:33,351 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/.tmp/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:33,353 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/.tmp/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7 empty. 2023-07-14 09:06:33,354 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/.tmp/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:33,354 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-14 09:06:33,386 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-14 09:06:33,392 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => fc5008608be0e2d76472fa37e4d943d7, NAME => 'hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/.tmp 2023-07-14 09:06:33,407 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 09:06:33,407 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing fc5008608be0e2d76472fa37e4d943d7, disabling compactions & flushes 2023-07-14 09:06:33,407 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:33,407 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:33,408 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. after waiting 0 ms 2023-07-14 09:06:33,408 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:33,408 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:33,408 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for fc5008608be0e2d76472fa37e4d943d7: 2023-07-14 09:06:33,412 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-14 09:06:33,428 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689325593415"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689325593415"}]},"ts":"1689325593415"} 2023-07-14 09:06:33,454 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-14 09:06:33,456 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-14 09:06:33,460 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689325593456"}]},"ts":"1689325593456"} 2023-07-14 09:06:33,465 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-14 09:06:33,469 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-07-14 09:06:33,471 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 09:06:33,471 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-14 09:06:33,471 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-14 09:06:33,473 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=fc5008608be0e2d76472fa37e4d943d7, ASSIGN}] 2023-07-14 09:06:33,476 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=fc5008608be0e2d76472fa37e4d943d7, ASSIGN 2023-07-14 09:06:33,479 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=fc5008608be0e2d76472fa37e4d943d7, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,33419,1689325591459; forceNewPlan=false, retain=false 2023-07-14 09:06:33,631 INFO [jenkins-hbase9:44817] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-14 09:06:33,632 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=fc5008608be0e2d76472fa37e4d943d7, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:33,633 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689325593632"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689325593632"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689325593632"}]},"ts":"1689325593632"} 2023-07-14 09:06:33,636 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure fc5008608be0e2d76472fa37e4d943d7, server=jenkins-hbase9.apache.org,33419,1689325591459}] 2023-07-14 09:06:33,790 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:33,790 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-14 09:06:33,793 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:39940, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-14 09:06:33,798 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:33,799 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fc5008608be0e2d76472fa37e4d943d7, NAME => 'hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7.', STARTKEY => '', ENDKEY => ''} 2023-07-14 09:06:33,800 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:33,800 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 09:06:33,800 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:33,800 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:33,803 INFO [StoreOpener-fc5008608be0e2d76472fa37e4d943d7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:33,805 DEBUG [StoreOpener-fc5008608be0e2d76472fa37e4d943d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/info 2023-07-14 09:06:33,805 DEBUG [StoreOpener-fc5008608be0e2d76472fa37e4d943d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/info 2023-07-14 09:06:33,806 INFO [StoreOpener-fc5008608be0e2d76472fa37e4d943d7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fc5008608be0e2d76472fa37e4d943d7 columnFamilyName info 2023-07-14 09:06:33,807 INFO [StoreOpener-fc5008608be0e2d76472fa37e4d943d7-1] regionserver.HStore(310): Store=fc5008608be0e2d76472fa37e4d943d7/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:33,808 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:33,809 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:33,814 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:33,818 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-14 09:06:33,819 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened fc5008608be0e2d76472fa37e4d943d7; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62513094, jitterRate=-0.06848230957984924}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 09:06:33,819 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for fc5008608be0e2d76472fa37e4d943d7: 2023-07-14 09:06:33,821 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7., pid=6, masterSystemTime=1689325593790 2023-07-14 09:06:33,825 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:33,825 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:33,827 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=fc5008608be0e2d76472fa37e4d943d7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:33,827 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689325593826"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689325593826"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689325593826"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689325593826"}]},"ts":"1689325593826"} 2023-07-14 09:06:33,835 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-14 09:06:33,835 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure fc5008608be0e2d76472fa37e4d943d7, server=jenkins-hbase9.apache.org,33419,1689325591459 in 195 msec 2023-07-14 09:06:33,839 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-14 09:06:33,839 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=fc5008608be0e2d76472fa37e4d943d7, ASSIGN in 362 msec 2023-07-14 09:06:33,841 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-14 09:06:33,841 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689325593841"}]},"ts":"1689325593841"} 2023-07-14 09:06:33,844 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-14 09:06:33,847 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-14 09:06:33,850 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 528 msec 2023-07-14 09:06:33,930 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-14 09:06:33,933 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-14 09:06:33,933 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 09:06:33,954 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 09:06:33,956 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:39954, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 09:06:33,969 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-14 09:06:33,986 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-14 09:06:33,991 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 29 msec 2023-07-14 09:06:34,002 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-14 09:06:34,012 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-14 09:06:34,018 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-07-14 09:06:34,027 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-14 09:06:34,033 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-14 09:06:34,033 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.509sec 2023-07-14 09:06:34,036 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-14 09:06:34,037 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-14 09:06:34,037 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-14 09:06:34,038 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,44817,1689325590279-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-14 09:06:34,039 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,44817,1689325590279-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-14 09:06:34,047 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-14 09:06:34,089 DEBUG [Listener at localhost/45201] zookeeper.ReadOnlyZKClient(139): Connect 0x511bead0 to 127.0.0.1:52519 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 09:06:34,094 DEBUG [Listener at localhost/45201] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4630e6e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 09:06:34,108 DEBUG [hconnection-0x485466e9-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 09:06:34,119 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:51314, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 09:06:34,129 INFO [Listener at localhost/45201] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase9.apache.org,44817,1689325590279 2023-07-14 09:06:34,141 DEBUG [Listener at localhost/45201] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-14 09:06:34,144 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:49644, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-14 09:06:34,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44817] master.HMaster$4(2112): Client=jenkins//172.31.2.10 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-14 09:06:34,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44817] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-14 09:06:34,160 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-14 09:06:34,162 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-14 09:06:34,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44817] master.MasterRpcServices(700): Client=jenkins//172.31.2.10 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-14 09:06:34,164 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/.tmp/data/default/t/b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:34,165 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/.tmp/data/default/t/b2ab292f4fc17b21d3c864d08ea57223 empty. 2023-07-14 09:06:34,167 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/.tmp/data/default/t/b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:34,167 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-14 09:06:34,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44817] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-14 09:06:34,185 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-14 09:06:34,187 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => b2ab292f4fc17b21d3c864d08ea57223, NAME => 't,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/.tmp 2023-07-14 09:06:34,202 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 09:06:34,203 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing b2ab292f4fc17b21d3c864d08ea57223, disabling compactions & flushes 2023-07-14 09:06:34,203 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:34,203 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:34,203 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. after waiting 0 ms 2023-07-14 09:06:34,203 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:34,203 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:34,203 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for b2ab292f4fc17b21d3c864d08ea57223: 2023-07-14 09:06:34,207 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-14 09:06:34,208 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689325594208"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689325594208"}]},"ts":"1689325594208"} 2023-07-14 09:06:34,211 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-14 09:06:34,212 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-14 09:06:34,212 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689325594212"}]},"ts":"1689325594212"} 2023-07-14 09:06:34,215 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-14 09:06:34,218 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-07-14 09:06:34,219 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 09:06:34,219 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-14 09:06:34,219 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-14 09:06:34,220 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=b2ab292f4fc17b21d3c864d08ea57223, ASSIGN}] 2023-07-14 09:06:34,221 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=b2ab292f4fc17b21d3c864d08ea57223, ASSIGN 2023-07-14 09:06:34,223 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=b2ab292f4fc17b21d3c864d08ea57223, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,44241,1689325591411; forceNewPlan=false, retain=false 2023-07-14 09:06:34,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44817] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-14 09:06:34,373 INFO [jenkins-hbase9:44817] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-14 09:06:34,374 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=b2ab292f4fc17b21d3c864d08ea57223, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,44241,1689325591411 2023-07-14 09:06:34,374 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689325594374"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689325594374"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689325594374"}]},"ts":"1689325594374"} 2023-07-14 09:06:34,378 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure b2ab292f4fc17b21d3c864d08ea57223, server=jenkins-hbase9.apache.org,44241,1689325591411}] 2023-07-14 09:06:34,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44817] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-14 09:06:34,539 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:34,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b2ab292f4fc17b21d3c864d08ea57223, NAME => 't,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.', STARTKEY => '', ENDKEY => ''} 2023-07-14 09:06:34,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:34,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 09:06:34,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:34,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:34,542 INFO [StoreOpener-b2ab292f4fc17b21d3c864d08ea57223-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:34,545 DEBUG [StoreOpener-b2ab292f4fc17b21d3c864d08ea57223-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/f 2023-07-14 09:06:34,545 DEBUG [StoreOpener-b2ab292f4fc17b21d3c864d08ea57223-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/f 2023-07-14 09:06:34,546 INFO [StoreOpener-b2ab292f4fc17b21d3c864d08ea57223-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b2ab292f4fc17b21d3c864d08ea57223 columnFamilyName f 2023-07-14 09:06:34,547 INFO [StoreOpener-b2ab292f4fc17b21d3c864d08ea57223-1] regionserver.HStore(310): Store=b2ab292f4fc17b21d3c864d08ea57223/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:34,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:34,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:34,565 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:34,569 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-14 09:06:34,570 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened b2ab292f4fc17b21d3c864d08ea57223; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62170009, jitterRate=-0.07359467446804047}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 09:06:34,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for b2ab292f4fc17b21d3c864d08ea57223: 2023-07-14 09:06:34,571 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223., pid=11, masterSystemTime=1689325594532 2023-07-14 09:06:34,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:34,576 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:34,576 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=b2ab292f4fc17b21d3c864d08ea57223, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,44241,1689325591411 2023-07-14 09:06:34,576 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689325594576"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689325594576"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689325594576"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689325594576"}]},"ts":"1689325594576"} 2023-07-14 09:06:34,586 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-14 09:06:34,586 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure b2ab292f4fc17b21d3c864d08ea57223, server=jenkins-hbase9.apache.org,44241,1689325591411 in 202 msec 2023-07-14 09:06:34,591 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-14 09:06:34,591 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=b2ab292f4fc17b21d3c864d08ea57223, ASSIGN in 366 msec 2023-07-14 09:06:34,592 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-14 09:06:34,593 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689325594593"}]},"ts":"1689325594593"} 2023-07-14 09:06:34,596 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-14 09:06:34,599 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-14 09:06:34,602 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 443 msec 2023-07-14 09:06:34,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44817] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-14 09:06:34,782 INFO [Listener at localhost/45201] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-14 09:06:34,782 DEBUG [Listener at localhost/45201] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-14 09:06:34,784 INFO [Listener at localhost/45201] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-14 09:06:34,788 INFO [Listener at localhost/45201] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-14 09:06:34,789 INFO [Listener at localhost/45201] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-14 09:06:34,789 INFO [Listener at localhost/45201] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-14 09:06:34,837 INFO [Listener at localhost/45201] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=292, OpenFileDescriptor=585, MaxFileDescriptor=60000, SystemLoadAverage=145, ProcessCount=174, AvailableMemoryMB=10250 2023-07-14 09:06:34,839 INFO [Listener at localhost/45201] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-14 09:06:34,842 DEBUG [Listener at localhost/45201] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-14 09:06:34,842 DEBUG [Listener at localhost/45201] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:34,843 INFO [Listener at localhost/45201] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-14 09:06:34,843 INFO [Listener at localhost/45201] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-14 09:06:34,846 INFO [Listener at localhost/45201] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689325593841/Put/vlen=2/seqid=0} 2023-07-14 09:06:34,846 INFO [Listener at localhost/45201] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-14 09:06:34,846 INFO [Listener at localhost/45201] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7./info:regioninfo/1689325593826/Put/vlen=41/seqid=0, hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7./info:seqnumDuringOpen/1689325593826/Put/vlen=8/seqid=0, hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7./info:server/1689325593826/Put/vlen=31/seqid=0, hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7./info:serverstartcode/1689325593826/Put/vlen=8/seqid=0, hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7./info:sn/1689325593632/Put/vlen=45/seqid=0, hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7./info:state/1689325593826/Put/vlen=4/seqid=0} 2023-07-14 09:06:34,846 INFO [Listener at localhost/45201] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => fc5008608be0e2d76472fa37e4d943d7, NAME => 'hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7.', STARTKEY => '', ENDKEY => ''} 2023-07-14 09:06:34,846 INFO [Listener at localhost/45201] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689325594593/Put/vlen=2/seqid=0} 2023-07-14 09:06:34,846 INFO [Listener at localhost/45201] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-14 09:06:34,847 INFO [Listener at localhost/45201] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223./info:regioninfo/1689325594576/Put/vlen=35/seqid=0, t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223./info:seqnumDuringOpen/1689325594576/Put/vlen=8/seqid=0, t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223./info:server/1689325594576/Put/vlen=31/seqid=0, t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223./info:serverstartcode/1689325594576/Put/vlen=8/seqid=0, t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223./info:sn/1689325594374/Put/vlen=45/seqid=0, t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223./info:state/1689325594576/Put/vlen=4/seqid=0} 2023-07-14 09:06:34,847 INFO [Listener at localhost/45201] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => b2ab292f4fc17b21d3c864d08ea57223, NAME => 't,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.', STARTKEY => '', ENDKEY => ''} 2023-07-14 09:06:34,848 DEBUG [Listener at localhost/45201] zookeeper.ReadOnlyZKClient(139): Connect 0x07944646 to 127.0.0.1:52519 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 09:06:34,852 DEBUG [Listener at localhost/45201] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@420100ae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 09:06:34,852 INFO [Listener at localhost/45201] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-14 09:06:34,852 INFO [Listener at localhost/45201] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-14 09:06:34,883 DEBUG [hconnection-0x31a3e955-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 09:06:34,889 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:51330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 09:06:34,894 INFO [Listener at localhost/45201] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-14 09:06:34,898 ERROR [Listener at localhost/45201] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase9.apache.org,44241,1689325591411: die! ***** 2023-07-14 09:06:34,899 ERROR [Listener at localhost/45201] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-14 09:06:34,902 DEBUG [Listener at localhost/45201] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-14 09:06:34,905 DEBUG [Listener at localhost/45201] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-14 09:06:34,905 DEBUG [Listener at localhost/45201] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-14 09:06:34,905 DEBUG [Listener at localhost/45201] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-14 09:06:34,906 INFO [Listener at localhost/45201] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 979894272, "init": 513802240, "max": 2051014656, "used": 142162208 }, "NonHeapMemoryUsage": { "committed": 113074176, "init": 2555904, "max": -1, "used": 110980176 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase9.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase9.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase9.apache.org" } ] } 2023-07-14 09:06:34,911 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=44817] master.MasterRpcServices(612): jenkins-hbase9.apache.org,44241,1689325591411 reported a fatal error: ***** ABORTING region server jenkins-hbase9.apache.org,44241,1689325591411: die! ***** 2023-07-14 09:06:34,913 INFO [Listener at localhost/45201] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,44241,1689325591411' ***** 2023-07-14 09:06:34,914 INFO [Listener at localhost/45201] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-14 09:06:34,914 INFO [RS:0;jenkins-hbase9:44241] regionserver.HeapMemoryManager(220): Stopping 2023-07-14 09:06:34,914 INFO [RS:0;jenkins-hbase9:44241] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-14 09:06:34,914 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-14 09:06:34,914 INFO [RS:0;jenkins-hbase9:44241] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-14 09:06:34,915 INFO [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer(3305): Received CLOSE for b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:34,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44241] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner 7439908509530980363 2023-07-14 09:06:34,915 INFO [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer(1141): aborting server jenkins-hbase9.apache.org,44241,1689325591411 2023-07-14 09:06:34,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44241] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,44241,1689325591411 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 09:06:34,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing b2ab292f4fc17b21d3c864d08ea57223, disabling compactions & flushes 2023-07-14 09:06:34,916 DEBUG [RS:0;jenkins-hbase9:44241] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x55da604d to 127.0.0.1:52519 2023-07-14 09:06:34,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44241] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.2.10:51330 deadline: 1689325654915, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,44241,1689325591411 aborting 2023-07-14 09:06:34,916 DEBUG [RS:0;jenkins-hbase9:44241] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 09:06:34,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:34,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:34,917 INFO [RS:0;jenkins-hbase9:44241] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-14 09:06:34,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. after waiting 0 ms 2023-07-14 09:06:34,917 INFO [RS:0;jenkins-hbase9:44241] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-14 09:06:34,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:34,917 INFO [RS:0;jenkins-hbase9:44241] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-14 09:06:34,917 INFO [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-14 09:06:34,917 INFO [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-07-14 09:06:34,917 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:34,918 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-14 09:06:34,918 DEBUG [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, b2ab292f4fc17b21d3c864d08ea57223=t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.} 2023-07-14 09:06:34,918 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-14 09:06:34,919 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:34,919 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-14 09:06:34,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for b2ab292f4fc17b21d3c864d08ea57223: 2023-07-14 09:06:34,919 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-14 09:06:34,919 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-14 09:06:34,919 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1825): Memstore data size is 2394 in region hbase:meta,,1.1588230740 2023-07-14 09:06:34,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:34,919 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-14 09:06:34,921 DEBUG [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-14 09:06:34,922 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-14 09:06:34,922 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-14 09:06:34,922 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-14 09:06:34,936 DEBUG [Listener at localhost/45201] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,44241,1689325591411 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,44241,1689325591411 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-14 09:06:34,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44241] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,44241,1689325591411 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 09:06:34,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44241] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.2.10:51330 deadline: 1689325654939, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,44241,1689325591411 aborting 2023-07-14 09:06:35,044 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=44241] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,44241,1689325591411 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2513) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44992) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 09:06:35,045 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=44241] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Get size: 76 connection: 172.31.2.10:51330 deadline: 1689325655044, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,44241,1689325591411 aborting 2023-07-14 09:06:35,121 INFO [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,44241,1689325591411; all regions closed. 2023-07-14 09:06:35,146 DEBUG [RS:0;jenkins-hbase9:44241] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 09:06:35,147 INFO [RS:0;jenkins-hbase9:44241] regionserver.LeaseManager(133): Closed leases 2023-07-14 09:06:35,147 INFO [RS:0;jenkins-hbase9:44241] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-14 09:06:35,148 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-14 09:06:35,148 INFO [RS:0;jenkins-hbase9:44241] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:44241 2023-07-14 09:06:35,149 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=44241] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,44241,1689325591411 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2513) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44992) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 09:06:35,150 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=44241] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Get size: 76 connection: 172.31.2.10:51330 deadline: 1689325655149, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,44241,1689325591411 aborting 2023-07-14 09:06:35,158 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:44241-0x101630797730001, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,44241,1689325591411 2023-07-14 09:06:35,158 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:33419-0x101630797730002, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,44241,1689325591411 2023-07-14 09:06:35,158 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 09:06:35,158 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:33419-0x101630797730002, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 09:06:35,158 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:44241-0x101630797730001, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 09:06:35,160 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,44241,1689325591411] 2023-07-14 09:06:35,160 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,44241,1689325591411; numProcessing=1 2023-07-14 09:06:35,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33419-0x101630797730002, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:35,161 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase9.apache.org,44241,1689325591411 znode expired, triggering replicatorRemoved event 2023-07-14 09:06:35,162 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,44241,1689325591411 already deleted, retry=false 2023-07-14 09:06:35,162 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase9.apache.org,44241,1689325591411 on jenkins-hbase9.apache.org,44817,1689325590279 2023-07-14 09:06:35,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33419-0x101630797730002, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:35,172 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase9.apache.org,44241,1689325591411, splitWal=true, meta=true 2023-07-14 09:06:35,172 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase9.apache.org,44241,1689325591411 (carryingMeta=true) jenkins-hbase9.apache.org,44241,1689325591411/CRASHED/regionCount=2/lock=java.util.concurrent.locks.ReentrantReadWriteLock@7fb0d192[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-14 09:06:35,175 INFO [PEWorker-4] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,44241,1689325591411, splitWal=true, meta=true 2023-07-14 09:06:35,176 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,44241,1689325591411, splitWal=true, meta=true, isMeta: true 2023-07-14 09:06:35,178 DEBUG [PEWorker-4] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting 2023-07-14 09:06:35,179 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase9.apache.org,44241,1689325591411 WAL count=1, meta=true 2023-07-14 09:06:35,183 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta}] 2023-07-14 09:06:35,190 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:35,192 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta, worker=jenkins-hbase9.apache.org,33419,1689325591459}] 2023-07-14 09:06:35,261 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:44241-0x101630797730001, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 09:06:35,261 INFO [RS:0;jenkins-hbase9:44241] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,44241,1689325591411; zookeeper connection closed. 2023-07-14 09:06:35,261 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:44241-0x101630797730001, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 09:06:35,261 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3f34d53c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3f34d53c 2023-07-14 09:06:35,354 WARN [RPCClient-NioEventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:44241 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:44241 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 09:06:35,355 DEBUG [RPCClient-NioEventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:44241 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:44241 2023-07-14 09:06:35,357 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33419] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-14 09:06:35,383 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta, size=3.1 K (3185bytes) 2023-07-14 09:06:35,384 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta 2023-07-14 09:06:35,390 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta after 6ms 2023-07-14 09:06:35,397 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(311): Open hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta took 16ms 2023-07-14 09:06:35,415 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-14 09:06:35,418 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta so closing down 2023-07-14 09:06:35,418 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-14 09:06:35,424 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-14 09:06:35,427 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta.temp 2023-07-14 09:06:35,429 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-14 09:06:35,471 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-14 09:06:35,474 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta.temp to hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-14 09:06:35,474 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 75 ms; skipped=1; WAL=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false 2023-07-14 09:06:35,475 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta, journal: Splitting hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta, size=3.1 K (3185bytes) at 1689325595383Finishing writing output for hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta so closing down at 1689325595418 (+35 ms)Creating recovered edits writer path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta.temp at 1689325595427 (+9 ms)3 split writer threads finished at 1689325595430 (+3 ms)Closed recovered edits writer path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689325595471 (+41 ms)Rename recovered edits hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta.temp to hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 at 1689325595474 (+3 ms)Processed 11 edits across 1 Regions in 75 ms; skipped=1; WAL=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false at 1689325595475 (+1 ms) 2023-07-14 09:06:35,475 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta 2023-07-14 09:06:35,478 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-14 09:06:35,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44817] master.HMaster(3641): Remote procedure done, pid=14 2023-07-14 09:06:35,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44817] wal.WALSplitUtil(141): Moved hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta to hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/oldWALs 2023-07-14 09:06:35,494 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-14 09:06:35,494 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta, worker=jenkins-hbase9.apache.org,33419,1689325591459 in 299 msec 2023-07-14 09:06:35,496 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:35,500 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-14 09:06:35,501 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase9.apache.org%2C44241%2C1689325591411.meta.1689325593033.meta, worker=jenkins-hbase9.apache.org,33419,1689325591459 in 316 msec 2023-07-14 09:06:35,504 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting dir is empty, no logs to split. 2023-07-14 09:06:35,504 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase9.apache.org,44241,1689325591411 WAL count=0, meta=true 2023-07-14 09:06:35,504 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase9.apache.org,44241,1689325591411 WAL splitting is done? wals=0, meta=true 2023-07-14 09:06:35,506 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-14 09:06:35,508 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-14 09:06:35,509 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 09:06:35,647 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-14 09:06:35,659 DEBUG [hconnection-0x31a3e955-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase9.apache.org/172.31.2.10:44241 this server is in the failed servers list 2023-07-14 09:06:35,659 DEBUG [jenkins-hbase9:44817] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-14 09:06:35,659 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,33419,1689325591459, state=OPENING 2023-07-14 09:06:35,662 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-14 09:06:35,662 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,33419,1689325591459}] 2023-07-14 09:06:35,662 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 09:06:35,821 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-14 09:06:35,821 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 09:06:35,824 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta, suffix=.meta, logDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459, archiveDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/oldWALs, maxLogs=32 2023-07-14 09:06:35,842 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41183,DS-06b9980b-5da4-4372-b119-090a73b88c97,DISK] 2023-07-14 09:06:35,843 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33565,DS-46c52e53-4cf7-418f-9976-a339137f48a3,DISK] 2023-07-14 09:06:35,845 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459/jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta 2023-07-14 09:06:35,845 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41183,DS-06b9980b-5da4-4372-b119-090a73b88c97,DISK], DatanodeInfoWithStorage[127.0.0.1:33565,DS-46c52e53-4cf7-418f-9976-a339137f48a3,DISK]] 2023-07-14 09:06:35,846 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-14 09:06:35,846 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-14 09:06:35,846 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-14 09:06:35,846 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-14 09:06:35,846 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-14 09:06:35,847 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 09:06:35,847 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-14 09:06:35,847 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-14 09:06:35,848 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-14 09:06:35,850 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/info 2023-07-14 09:06:35,850 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/info 2023-07-14 09:06:35,850 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-14 09:06:35,851 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:35,851 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-14 09:06:35,852 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/rep_barrier 2023-07-14 09:06:35,852 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/rep_barrier 2023-07-14 09:06:35,853 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-14 09:06:35,854 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:35,854 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-14 09:06:35,855 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/table 2023-07-14 09:06:35,855 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/table 2023-07-14 09:06:35,856 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-14 09:06:35,856 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:35,858 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740 2023-07-14 09:06:35,860 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740 2023-07-14 09:06:35,861 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-14 09:06:35,869 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5464): Applied 24, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-14 09:06:35,870 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=2.34 KB heapSize=4.86 KB 2023-07-14 09:06:35,920 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.18 KB at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/.tmp/info/0fc741c44d8847e585b28c4f1f4ce995 2023-07-14 09:06:35,975 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/.tmp/table/c97c898f88734748bec14934376d6f27 2023-07-14 09:06:35,983 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/.tmp/info/0fc741c44d8847e585b28c4f1f4ce995 as hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/info/0fc741c44d8847e585b28c4f1f4ce995 2023-07-14 09:06:35,991 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/info/0fc741c44d8847e585b28c4f1f4ce995, entries=20, sequenceid=13, filesize=6.9 K 2023-07-14 09:06:35,994 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/.tmp/table/c97c898f88734748bec14934376d6f27 as hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/table/c97c898f88734748bec14934376d6f27 2023-07-14 09:06:36,000 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/table/c97c898f88734748bec14934376d6f27, entries=4, sequenceid=13, filesize=4.7 K 2023-07-14 09:06:36,001 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.34 KB/2394, heapSize ~4.58 KB/4688, currentSize=0 B/0 for 1588230740 in 132ms, sequenceid=13, compaction requested=false; wal=null 2023-07-14 09:06:36,002 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-14 09:06:36,005 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-14 09:06:36,008 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-14 09:06:36,010 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-14 09:06:36,013 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-14 09:06:36,014 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68080338, jitterRate=0.014476090669631958}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-14 09:06:36,014 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-14 09:06:36,015 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=16, masterSystemTime=1689325595816 2023-07-14 09:06:36,018 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-14 09:06:36,018 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-14 09:06:36,018 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,33419,1689325591459, state=OPEN 2023-07-14 09:06:36,020 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-14 09:06:36,020 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 09:06:36,022 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-14 09:06:36,023 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,33419,1689325591459 in 358 msec 2023-07-14 09:06:36,025 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-14 09:06:36,025 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 516 msec 2023-07-14 09:06:36,025 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase9.apache.org,44241,1689325591411 had 2 regions 2023-07-14 09:06:36,027 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,44241,1689325591411, splitWal=true, meta=true, isMeta: false 2023-07-14 09:06:36,029 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase9.apache.org,44241,1689325591411 WAL count=1, meta=false 2023-07-14 09:06:36,029 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810}] 2023-07-14 09:06:36,031 DEBUG [PEWorker-1] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:36,033 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810, worker=jenkins-hbase9.apache.org,33419,1689325591459}] 2023-07-14 09:06:36,164 DEBUG [hconnection-0x31a3e955-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 09:06:36,168 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:39958, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 09:06:36,185 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33419] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=18 2023-07-14 09:06:36,186 DEBUG [hconnection-0x31a3e955-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase9.apache.org/172.31.2.10:44241 this server is in the failed servers list 2023-07-14 09:06:36,198 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810, size=1.1 K (1148bytes) 2023-07-14 09:06:36,199 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810 2023-07-14 09:06:36,199 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810 after 0ms 2023-07-14 09:06:36,202 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(311): Open hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810 took 4ms 2023-07-14 09:06:36,205 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(346): Last flushed sequenceid for b2ab292f4fc17b21d3c864d08ea57223: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-14 09:06:36,207 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810 so closing down 2023-07-14 09:06:36,207 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-14 09:06:36,213 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810.temp 2023-07-14 09:06:36,214 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-14 09:06:36,226 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-14 09:06:36,228 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810.temp to hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/recovered.edits/0000000000000000013 2023-07-14 09:06:36,228 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 25 ms; skipped=1; WAL=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810, size=1.1 K, length=1148, corrupted=false, cancelled=false 2023-07-14 09:06:36,228 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810, journal: Splitting hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810, size=1.1 K (1148bytes) at 1689325596199Finishing writing output for hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810 so closing down at 1689325596207 (+8 ms)Creating recovered edits writer path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810.temp at 1689325596214 (+7 ms)3 split writer threads finished at 1689325596214Closed recovered edits writer path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689325596226 (+12 ms)Rename recovered edits hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810.temp to hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/recovered.edits/0000000000000000013 at 1689325596228 (+2 ms)Processed 11 edits across 1 Regions in 25 ms; skipped=1; WAL=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810, size=1.1 K, length=1148, corrupted=false, cancelled=false at 1689325596228 2023-07-14 09:06:36,228 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810 2023-07-14 09:06:36,228 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=18 2023-07-14 09:06:36,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44817] master.HMaster(3641): Remote procedure done, pid=18 2023-07-14 09:06:36,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44817] wal.WALSplitUtil(141): Moved hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting/jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810 to hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/oldWALs 2023-07-14 09:06:36,235 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-14 09:06:36,235 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810, worker=jenkins-hbase9.apache.org,33419,1689325591459 in 200 msec 2023-07-14 09:06:36,237 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:36,239 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=12 2023-07-14 09:06:36,239 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase9.apache.org%2C44241%2C1689325591411.1689325592810, worker=jenkins-hbase9.apache.org,33419,1689325591459 in 208 msec 2023-07-14 09:06:36,241 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,44241,1689325591411-splitting dir is empty, no logs to split. 2023-07-14 09:06:36,241 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase9.apache.org,44241,1689325591411 WAL count=0, meta=false 2023-07-14 09:06:36,241 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase9.apache.org,44241,1689325591411 WAL splitting is done? wals=0, meta=false 2023-07-14 09:06:36,243 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=b2ab292f4fc17b21d3c864d08ea57223, ASSIGN}] 2023-07-14 09:06:36,244 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=b2ab292f4fc17b21d3c864d08ea57223, ASSIGN 2023-07-14 09:06:36,245 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=b2ab292f4fc17b21d3c864d08ea57223, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 09:06:36,393 DEBUG [hconnection-0x31a3e955-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase9.apache.org/172.31.2.10:44241 this server is in the failed servers list 2023-07-14 09:06:36,396 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=b2ab292f4fc17b21d3c864d08ea57223, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:36,396 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689325596396"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689325596396"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689325596396"}]},"ts":"1689325596396"} 2023-07-14 09:06:36,397 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:44241 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:44241 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 09:06:36,398 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:44241 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:44241 2023-07-14 09:06:36,502 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE; OpenRegionProcedure b2ab292f4fc17b21d3c864d08ea57223, server=jenkins-hbase9.apache.org,33419,1689325591459}] 2023-07-14 09:06:36,660 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:36,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b2ab292f4fc17b21d3c864d08ea57223, NAME => 't,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.', STARTKEY => '', ENDKEY => ''} 2023-07-14 09:06:36,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:36,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 09:06:36,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:36,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:36,663 INFO [StoreOpener-b2ab292f4fc17b21d3c864d08ea57223-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:36,664 DEBUG [StoreOpener-b2ab292f4fc17b21d3c864d08ea57223-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/f 2023-07-14 09:06:36,664 DEBUG [StoreOpener-b2ab292f4fc17b21d3c864d08ea57223-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/f 2023-07-14 09:06:36,665 INFO [StoreOpener-b2ab292f4fc17b21d3c864d08ea57223-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b2ab292f4fc17b21d3c864d08ea57223 columnFamilyName f 2023-07-14 09:06:36,666 INFO [StoreOpener-b2ab292f4fc17b21d3c864d08ea57223-1] regionserver.HStore(310): Store=b2ab292f4fc17b21d3c864d08ea57223/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:36,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:36,670 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:36,671 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/recovered.edits/0000000000000000013 2023-07-14 09:06:36,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/recovered.edits/0000000000000000013 2023-07-14 09:06:36,675 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing b2ab292f4fc17b21d3c864d08ea57223 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-14 09:06:36,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/.tmp/f/41bb6c12e40f43ad912f1d7c2fff5c75 2023-07-14 09:06:36,699 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/.tmp/f/41bb6c12e40f43ad912f1d7c2fff5c75 as hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/f/41bb6c12e40f43ad912f1d7c2fff5c75 2023-07-14 09:06:36,701 DEBUG [hconnection-0x31a3e955-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase9.apache.org/172.31.2.10:44241 this server is in the failed servers list 2023-07-14 09:06:36,708 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/f/41bb6c12e40f43ad912f1d7c2fff5c75, entries=10, sequenceid=13, filesize=4.8 K 2023-07-14 09:06:36,708 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for b2ab292f4fc17b21d3c864d08ea57223 in 33ms, sequenceid=13, compaction requested=false; wal=null 2023-07-14 09:06:36,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-14 09:06:36,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/recovered.edits/0000000000000000013 2023-07-14 09:06:36,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:36,717 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-14 09:06:36,718 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened b2ab292f4fc17b21d3c864d08ea57223; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61627305, jitterRate=-0.08168159425258636}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 09:06:36,718 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for b2ab292f4fc17b21d3c864d08ea57223: 2023-07-14 09:06:36,719 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223., pid=20, masterSystemTime=1689325596655 2023-07-14 09:06:36,721 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:36,721 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:36,722 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=b2ab292f4fc17b21d3c864d08ea57223, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:36,722 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689325596722"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689325596722"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689325596722"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689325596722"}]},"ts":"1689325596722"} 2023-07-14 09:06:36,727 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-07-14 09:06:36,727 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; OpenRegionProcedure b2ab292f4fc17b21d3c864d08ea57223, server=jenkins-hbase9.apache.org,33419,1689325591459 in 222 msec 2023-07-14 09:06:36,730 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=12 2023-07-14 09:06:36,730 INFO [PEWorker-1] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase9.apache.org,44241,1689325591411 after splitting done 2023-07-14 09:06:36,730 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=b2ab292f4fc17b21d3c864d08ea57223, ASSIGN in 484 msec 2023-07-14 09:06:36,730 DEBUG [PEWorker-1] master.DeadServer(114): Removed jenkins-hbase9.apache.org,44241,1689325591411 from processing; numProcessing=0 2023-07-14 09:06:36,732 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase9.apache.org,44241,1689325591411, splitWal=true, meta=true in 1.5660 sec 2023-07-14 09:06:37,215 DEBUG [Listener at localhost/45201] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07944646 to 127.0.0.1:52519 2023-07-14 09:06:37,215 DEBUG [Listener at localhost/45201] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 09:06:37,217 INFO [Listener at localhost/45201] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-14 09:06:37,230 INFO [Listener at localhost/45201] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=266 (was 292), OpenFileDescriptor=569 (was 585), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=134 (was 145), ProcessCount=174 (was 174), AvailableMemoryMB=10198 (was 10250) 2023-07-14 09:06:37,241 INFO [Listener at localhost/45201] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=266, OpenFileDescriptor=569, MaxFileDescriptor=60000, SystemLoadAverage=134, ProcessCount=174, AvailableMemoryMB=10197 2023-07-14 09:06:37,241 INFO [Listener at localhost/45201] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase9:44241 2023-07-14 09:06:37,254 INFO [Listener at localhost/45201] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-07-14 09:06:37,254 INFO [Listener at localhost/45201] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 09:06:37,254 INFO [Listener at localhost/45201] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-14 09:06:37,254 INFO [Listener at localhost/45201] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-14 09:06:37,254 INFO [Listener at localhost/45201] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 09:06:37,254 INFO [Listener at localhost/45201] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-14 09:06:37,255 INFO [Listener at localhost/45201] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-14 09:06:37,256 INFO [Listener at localhost/45201] ipc.NettyRpcServer(120): Bind to /172.31.2.10:34111 2023-07-14 09:06:37,257 INFO [Listener at localhost/45201] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-14 09:06:37,258 DEBUG [Listener at localhost/45201] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-14 09:06:37,258 INFO [Listener at localhost/45201] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 09:06:37,260 INFO [Listener at localhost/45201] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 09:06:37,261 INFO [Listener at localhost/45201] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34111 connecting to ZooKeeper ensemble=127.0.0.1:52519 2023-07-14 09:06:37,264 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:341110x0, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-14 09:06:37,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:34111-0x101630797730008 connected 2023-07-14 09:06:37,265 DEBUG [Listener at localhost/45201] zookeeper.ZKUtil(162): regionserver:34111-0x101630797730008, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-14 09:06:37,266 DEBUG [Listener at localhost/45201] zookeeper.ZKUtil(162): regionserver:34111-0x101630797730008, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-14 09:06:37,267 DEBUG [Listener at localhost/45201] zookeeper.ZKUtil(164): regionserver:34111-0x101630797730008, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-14 09:06:37,268 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34111 2023-07-14 09:06:37,269 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34111 2023-07-14 09:06:37,269 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34111 2023-07-14 09:06:37,269 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34111 2023-07-14 09:06:37,270 DEBUG [Listener at localhost/45201] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34111 2023-07-14 09:06:37,272 INFO [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(951): ClusterId : 770ab0a7-e7c2-47a2-ae74-1d796eeb3b23 2023-07-14 09:06:37,272 DEBUG [RS:2;jenkins-hbase9:34111] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-14 09:06:37,274 DEBUG [RS:2;jenkins-hbase9:34111] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-14 09:06:37,274 DEBUG [RS:2;jenkins-hbase9:34111] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-14 09:06:37,277 DEBUG [RS:2;jenkins-hbase9:34111] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-14 09:06:37,278 DEBUG [RS:2;jenkins-hbase9:34111] zookeeper.ReadOnlyZKClient(139): Connect 0x407d9f81 to 127.0.0.1:52519 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 09:06:37,288 DEBUG [RS:2;jenkins-hbase9:34111] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4eba79fe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 09:06:37,288 DEBUG [RS:2;jenkins-hbase9:34111] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a85f4f1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-14 09:06:37,296 DEBUG [RS:2;jenkins-hbase9:34111] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase9:34111 2023-07-14 09:06:37,297 INFO [RS:2;jenkins-hbase9:34111] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-14 09:06:37,297 INFO [RS:2;jenkins-hbase9:34111] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-14 09:06:37,297 DEBUG [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(1022): About to register with Master. 2023-07-14 09:06:37,297 INFO [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase9.apache.org,44817,1689325590279 with isa=jenkins-hbase9.apache.org/172.31.2.10:34111, startcode=1689325597253 2023-07-14 09:06:37,297 DEBUG [RS:2;jenkins-hbase9:34111] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-14 09:06:37,300 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:46103, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-14 09:06:37,300 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44817] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:37,301 DEBUG [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4 2023-07-14 09:06:37,301 DEBUG [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:46473 2023-07-14 09:06:37,301 DEBUG [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-14 09:06:37,302 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 09:06:37,302 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:33419-0x101630797730002, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 09:06:37,303 DEBUG [RS:2;jenkins-hbase9:34111] zookeeper.ZKUtil(162): regionserver:34111-0x101630797730008, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:37,303 WARN [RS:2;jenkins-hbase9:34111] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-14 09:06:37,303 INFO [RS:2;jenkins-hbase9:34111] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 09:06:37,303 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,34111,1689325597253] 2023-07-14 09:06:37,303 DEBUG [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(1948): logDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:37,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33419-0x101630797730002, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:37,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33419-0x101630797730002, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:37,308 DEBUG [RS:2;jenkins-hbase9:34111] zookeeper.ZKUtil(162): regionserver:34111-0x101630797730008, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:37,308 DEBUG [RS:2;jenkins-hbase9:34111] zookeeper.ZKUtil(162): regionserver:34111-0x101630797730008, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:37,310 DEBUG [RS:2;jenkins-hbase9:34111] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-14 09:06:37,310 INFO [RS:2;jenkins-hbase9:34111] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-14 09:06:37,312 INFO [RS:2;jenkins-hbase9:34111] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-14 09:06:37,315 INFO [RS:2;jenkins-hbase9:34111] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-14 09:06:37,315 INFO [RS:2;jenkins-hbase9:34111] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:37,318 INFO [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-14 09:06:37,319 INFO [RS:2;jenkins-hbase9:34111] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:37,320 DEBUG [RS:2;jenkins-hbase9:34111] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:37,320 DEBUG [RS:2;jenkins-hbase9:34111] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:37,320 DEBUG [RS:2;jenkins-hbase9:34111] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:37,320 DEBUG [RS:2;jenkins-hbase9:34111] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:37,320 DEBUG [RS:2;jenkins-hbase9:34111] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:37,320 DEBUG [RS:2;jenkins-hbase9:34111] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-14 09:06:37,320 DEBUG [RS:2;jenkins-hbase9:34111] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:37,320 DEBUG [RS:2;jenkins-hbase9:34111] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:37,320 DEBUG [RS:2;jenkins-hbase9:34111] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:37,320 DEBUG [RS:2;jenkins-hbase9:34111] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 09:06:37,321 INFO [RS:2;jenkins-hbase9:34111] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:37,321 INFO [RS:2;jenkins-hbase9:34111] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:37,322 INFO [RS:2;jenkins-hbase9:34111] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:37,337 INFO [RS:2;jenkins-hbase9:34111] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-14 09:06:37,337 INFO [RS:2;jenkins-hbase9:34111] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,34111,1689325597253-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 09:06:37,348 INFO [RS:2;jenkins-hbase9:34111] regionserver.Replication(203): jenkins-hbase9.apache.org,34111,1689325597253 started 2023-07-14 09:06:37,348 INFO [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(1637): Serving as jenkins-hbase9.apache.org,34111,1689325597253, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:34111, sessionid=0x101630797730008 2023-07-14 09:06:37,348 INFO [Listener at localhost/45201] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase9:34111,5,FailOnTimeoutGroup] 2023-07-14 09:06:37,348 DEBUG [RS:2;jenkins-hbase9:34111] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-14 09:06:37,348 DEBUG [RS:2;jenkins-hbase9:34111] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:37,348 DEBUG [RS:2;jenkins-hbase9:34111] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,34111,1689325597253' 2023-07-14 09:06:37,348 INFO [Listener at localhost/45201] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase9.apache.org,44241,1689325591411 2023-07-14 09:06:37,348 DEBUG [RS:2;jenkins-hbase9:34111] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-14 09:06:37,348 INFO [Listener at localhost/45201] client.TestScannerTimeout(177): START ************ test3686b 2023-07-14 09:06:37,349 DEBUG [RS:2;jenkins-hbase9:34111] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-14 09:06:37,349 DEBUG [RS:2;jenkins-hbase9:34111] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-14 09:06:37,350 DEBUG [RS:2;jenkins-hbase9:34111] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-14 09:06:37,350 DEBUG [RS:2;jenkins-hbase9:34111] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:37,350 DEBUG [RS:2;jenkins-hbase9:34111] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,34111,1689325597253' 2023-07-14 09:06:37,350 DEBUG [RS:2;jenkins-hbase9:34111] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-14 09:06:37,350 WARN [RPCClient-NioEventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:44241 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:44241 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 09:06:37,350 DEBUG [RPCClient-NioEventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:44241 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:44241 2023-07-14 09:06:37,350 DEBUG [RS:2;jenkins-hbase9:34111] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-14 09:06:37,351 DEBUG [RS:2;jenkins-hbase9:34111] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-14 09:06:37,352 INFO [RS:2;jenkins-hbase9:34111] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-14 09:06:37,352 INFO [RS:2;jenkins-hbase9:34111] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-14 09:06:37,453 DEBUG [hconnection-0x485466e9-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 09:06:37,454 INFO [RS:2;jenkins-hbase9:34111] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C34111%2C1689325597253, suffix=, logDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,34111,1689325597253, archiveDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/oldWALs, maxLogs=32 2023-07-14 09:06:37,456 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:39962, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 09:06:37,459 DEBUG [Listener at localhost/45201] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-14 09:06:37,459 DEBUG [Listener at localhost/45201] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:37,460 DEBUG [Listener at localhost/45201] zookeeper.ReadOnlyZKClient(139): Connect 0x1d5176de to 127.0.0.1:52519 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 09:06:37,465 DEBUG [Listener at localhost/45201] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2bbe7d10, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 09:06:37,469 DEBUG [hconnection-0x6ee5f3a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 09:06:37,471 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:39968, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 09:06:37,481 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33565,DS-46c52e53-4cf7-418f-9976-a339137f48a3,DISK] 2023-07-14 09:06:37,481 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41183,DS-06b9980b-5da4-4372-b119-090a73b88c97,DISK] 2023-07-14 09:06:37,484 INFO [RS:2;jenkins-hbase9:34111] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,34111,1689325597253/jenkins-hbase9.apache.org%2C34111%2C1689325597253.1689325597457 2023-07-14 09:06:37,484 DEBUG [RS:2;jenkins-hbase9:34111] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33565,DS-46c52e53-4cf7-418f-9976-a339137f48a3,DISK], DatanodeInfoWithStorage[127.0.0.1:41183,DS-06b9980b-5da4-4372-b119-090a73b88c97,DISK]] 2023-07-14 09:06:38,633 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-14 09:06:38,692 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-14 09:06:40,434 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-14 09:06:40,434 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-14 09:06:52,481 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease 6294983190352232457 expired clientIPAndPort=172.31.2.10:39968, userName=jenkins, regionInfo=t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:54,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33419] regionserver.RSRpcServices(3094): Client tried to access missing scanner 6294983190352232457 2023-07-14 09:06:54,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33419] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.2.10:39968 deadline: 1689325674482, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '6294983190352232457'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-14 09:06:54,484 DEBUG [Listener at localhost/45201] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '6294983190352232457'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '6294983190352232457'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '6294983190352232457'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-14 09:06:54,487 DEBUG [Listener at localhost/45201] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d5176de to 127.0.0.1:52519 2023-07-14 09:06:54,487 DEBUG [Listener at localhost/45201] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 09:06:54,487 INFO [Listener at localhost/45201] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-14 09:06:54,502 INFO [Listener at localhost/45201] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=294 (was 266) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34111 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=34111 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:52519@0x1d5176de java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1149355354.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:52519@0x1d5176de-SendThread(127.0.0.1:52519) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34111 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=34111 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase9:34111 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:52519@0x407d9f81-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RS:2;jenkins-hbase9:34111-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1078391276_17 at /127.0.0.1:54918 [Receiving block BP-514055984-172.31.2.10-1689325587359:blk_1073741845_1021] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:46473 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34111 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-514055984-172.31.2.10-1689325587359:blk_1073741845_1021, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-514055984-172.31.2.10-1689325587359:blk_1073741845_1021, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34111 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34111 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4-prefix:jenkins-hbase9.apache.org,34111,1689325597253 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1078391276_17 at /127.0.0.1:32874 [Receiving block BP-514055984-172.31.2.10-1689325587359:blk_1073741845_1021] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: jenkins-hbase9:34111Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/cluster_708bca9d-6cdf-4da8-0283-1300549de5a3/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/cluster_708bca9d-6cdf-4da8-0283-1300549de5a3/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=34111 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:52519@0x407d9f81 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1149355354.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=34111 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:52519@0x1d5176de-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34111 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:52519@0x407d9f81-SendThread(127.0.0.1:52519) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=573 (was 569) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=118 (was 134), ProcessCount=174 (was 174), AvailableMemoryMB=10183 (was 10197) 2023-07-14 09:06:54,515 INFO [Listener at localhost/45201] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=294, OpenFileDescriptor=573, MaxFileDescriptor=60000, SystemLoadAverage=118, ProcessCount=174, AvailableMemoryMB=10183 2023-07-14 09:06:54,515 INFO [Listener at localhost/45201] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase9:44241 2023-07-14 09:06:54,515 INFO [Listener at localhost/45201] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase9.apache.org,44241,1689325591411 2023-07-14 09:06:54,515 INFO [Listener at localhost/45201] client.TestScannerTimeout(104): START************ test2772 2023-07-14 09:06:54,518 DEBUG [Listener at localhost/45201] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-14 09:06:54,519 DEBUG [Listener at localhost/45201] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:54,519 DEBUG [Listener at localhost/45201] zookeeper.ReadOnlyZKClient(139): Connect 0x7a742b87 to 127.0.0.1:52519 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 09:06:54,523 DEBUG [Listener at localhost/45201] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e93565a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 09:06:54,523 ERROR [Listener at localhost/45201] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase9.apache.org,33419,1689325591459: die! ***** 2023-07-14 09:06:54,523 ERROR [Listener at localhost/45201] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-14 09:06:54,524 DEBUG [Listener at localhost/45201] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-14 09:06:54,524 DEBUG [Listener at localhost/45201] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-14 09:06:54,529 DEBUG [Listener at localhost/45201] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-14 09:06:54,531 DEBUG [Listener at localhost/45201] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-14 09:06:54,545 INFO [Listener at localhost/45201] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 979894272, "init": 513802240, "max": 2051014656, "used": 280537336 }, "NonHeapMemoryUsage": { "committed": 123953152, "init": 2555904, "max": -1, "used": 121558384 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase9.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 12788, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 4, "ProcessCallTime_num_ops": 63, "ProcessCallTime_min": 0, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 0, "ProcessCallTime_25th_percentile": 0, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 63, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 63, "TotalCallTime_min": 0, "TotalCallTime_max": 1, "TotalCallTime_mean": 0, "TotalCallTime_25th_percentile": 0, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 63, "ResponseSize_min": 196, "ResponseSize_max": 678, "ResponseSize_mean": 437, "ResponseSize_25th_percentile": 196, "ResponseSize_median": 196, "ResponseSize_75th_percentile": 675, "ResponseSize_90th_percentile": 677, "ResponseSize_95th_percentile": 677, "ResponseSize_98th_percentile": 677, "ResponseSize_99th_percentile": 677, "ResponseSize_99.9th_percentile": 677, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 5, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 63, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 17483 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase9.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:52519", "tag.serverName": "jenkins-hbase9.apache.org,33419,1689325591459", "tag.clusterId": "770ab0a7-e7c2-47a2-ae74-1d796eeb3b23", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase9.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 3, "maxStoreFileCount": 1, "memStoreSize": 912, "storeFileSize": 16785, "maxStoreFileAge": 16690, "minStoreFileAge": 15920, "avgStoreFileAge": 16414, "numReferenceFiles": 0, "regionServerStartTime": 1689325591459, "averageRegionSize": 5899, "storeFileIndexSize": 1048, "staticIndexSize": 184, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 1787, "localBytesRead": 1787, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819799248, "blockCacheCount": 3, "blockCacheDataBlockCount": 3, "blockCacheSize": 606640, "blockCacheCountHitPercent": 87.5, "blockCacheExpressHitPercent": 87.5, "l1CacheSize": 606640, "l1CacheFreeSize": 819799248, "l1CacheCount": 3, "l1CacheEvictionCount": 0, "l1CacheHitCount": 21, "l1CacheMissCount": 3, "l1CacheHitRatio": 0.875, "l1CacheMissRatio": 0.125, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 7076, "ByteBuffAllocatorPoolAllocationBytes": 1264640, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 24, "totalRowActionRequestCount": 30, "readRequestCount": 26, "filteredReadRequestCount": 0, "writeRequestCount": 4, "rpcGetRequestCount": 8, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 12, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 4, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 21, "blockCacheHitCountPrimary": 21, "blockCacheHitCachingCount": 21, "blockCacheMissCount": 3, "blockCacheMissCountPrimary": 3, "blockCacheMissCachingCount": 3, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 3, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 21, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 34, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 2664, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 11, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 4, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 2, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 8, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 16785, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 2, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 11, "ScanSize_min": 155, "ScanSize_max": 653, "ScanSize_mean": 404, "ScanSize_25th_percentile": 155, "ScanSize_median": 156, "ScanSize_75th_percentile": 652, "ScanSize_90th_percentile": 652, "ScanSize_95th_percentile": 652, "ScanSize_98th_percentile": 652, "ScanSize_99th_percentile": 652, "ScanSize_99.9th_percentile": 652, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 2664, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 2, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.4641628607523356, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 24, "ServerWriteQueryPerSecond_mean_rate": 1.0951836518921243, "ServerWriteQueryPerSecond_1min_rate": 3.738243758742744, "ServerWriteQueryPerSecond_5min_rate": 4.565901237603426, "ServerWriteQueryPerSecond_15min_rate": 4.720662978343764 } ] } 2023-07-14 09:06:54,546 WARN [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=44817] master.MasterRpcServices(612): jenkins-hbase9.apache.org,33419,1689325591459 reported a fatal error: ***** ABORTING region server jenkins-hbase9.apache.org,33419,1689325591459: die! ***** 2023-07-14 09:06:54,547 INFO [Listener at localhost/45201] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,33419,1689325591459' ***** 2023-07-14 09:06:54,549 INFO [Listener at localhost/45201] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-14 09:06:54,549 INFO [RS:1;jenkins-hbase9:33419] regionserver.HeapMemoryManager(220): Stopping 2023-07-14 09:06:54,550 INFO [RS:1;jenkins-hbase9:33419] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-14 09:06:54,550 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-14 09:06:54,550 INFO [RS:1;jenkins-hbase9:33419] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-14 09:06:54,550 INFO [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(3305): Received CLOSE for b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:54,550 INFO [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(3305): Received CLOSE for fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:54,550 INFO [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(1141): aborting server jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:54,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing b2ab292f4fc17b21d3c864d08ea57223, disabling compactions & flushes 2023-07-14 09:06:54,551 DEBUG [RS:1;jenkins-hbase9:33419] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62aad9af to 127.0.0.1:52519 2023-07-14 09:06:54,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:54,551 DEBUG [RS:1;jenkins-hbase9:33419] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 09:06:54,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:54,551 INFO [RS:1;jenkins-hbase9:33419] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-14 09:06:54,551 INFO [RS:1;jenkins-hbase9:33419] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-14 09:06:54,551 INFO [RS:1;jenkins-hbase9:33419] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-14 09:06:54,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. after waiting 0 ms 2023-07-14 09:06:54,551 INFO [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-14 09:06:54,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:54,552 INFO [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-14 09:06:54,552 DEBUG [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, b2ab292f4fc17b21d3c864d08ea57223=t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223., fc5008608be0e2d76472fa37e4d943d7=hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7.} 2023-07-14 09:06:54,552 DEBUG [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(1504): Waiting on 1588230740, b2ab292f4fc17b21d3c864d08ea57223, fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:54,552 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-14 09:06:54,552 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-14 09:06:54,553 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-14 09:06:54,553 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-14 09:06:54,553 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-14 09:06:54,554 DEBUG [hconnection-0x5587207f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 09:06:54,558 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:44982, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 09:06:54,558 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33419] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,33419,1689325591459 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 09:06:54,559 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33419] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.2.10:44982 deadline: 1689325674558, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,33419,1689325591459 aborting 2023-07-14 09:06:54,560 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:54,560 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for b2ab292f4fc17b21d3c864d08ea57223: 2023-07-14 09:06:54,561 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1825): Memstore data size is 834 in region hbase:meta,,1.1588230740 2023-07-14 09:06:54,561 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-14 09:06:54,561 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:54,561 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-14 09:06:54,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing fc5008608be0e2d76472fa37e4d943d7, disabling compactions & flushes 2023-07-14 09:06:54,562 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-14 09:06:54,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:54,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:54,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. after waiting 0 ms 2023-07-14 09:06:54,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:54,562 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-14 09:06:54,562 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:54,563 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:54,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for fc5008608be0e2d76472fa37e4d943d7: 2023-07-14 09:06:54,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:54,663 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33419] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,33419,1689325591459 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 09:06:54,663 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33419] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.2.10:44982 deadline: 1689325674663, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,33419,1689325591459 aborting 2023-07-14 09:06:54,739 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-14 09:06:54,739 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-14 09:06:54,752 INFO [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,33419,1689325591459; all regions closed. 2023-07-14 09:06:54,763 DEBUG [RS:1;jenkins-hbase9:33419] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 09:06:54,763 INFO [RS:1;jenkins-hbase9:33419] regionserver.LeaseManager(133): Closed leases 2023-07-14 09:06:54,764 INFO [RS:1;jenkins-hbase9:33419] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-14 09:06:54,764 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-14 09:06:54,765 INFO [RS:1;jenkins-hbase9:33419] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:33419 2023-07-14 09:06:54,769 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:33419-0x101630797730002, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:54,769 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 09:06:54,769 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:34111-0x101630797730008, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33419,1689325591459 2023-07-14 09:06:54,769 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:34111-0x101630797730008, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 09:06:54,769 ERROR [Listener at localhost/45201-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3f6a55b3 rejected from java.util.concurrent.ThreadPoolExecutor@3b55c686[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1374) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:603) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-07-14 09:06:54,770 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:33419-0x101630797730002, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 09:06:54,771 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,33419,1689325591459] 2023-07-14 09:06:54,771 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,33419,1689325591459; numProcessing=1 2023-07-14 09:06:54,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34111-0x101630797730008, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:54,772 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase9.apache.org,33419,1689325591459 znode expired, triggering replicatorRemoved event 2023-07-14 09:06:54,772 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,33419,1689325591459 already deleted, retry=false 2023-07-14 09:06:54,772 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase9.apache.org,33419,1689325591459 on jenkins-hbase9.apache.org,44817,1689325590279 2023-07-14 09:06:54,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34111-0x101630797730008, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:54,774 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=21, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase9.apache.org,33419,1689325591459, splitWal=true, meta=true 2023-07-14 09:06:54,774 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=21 for jenkins-hbase9.apache.org,33419,1689325591459 (carryingMeta=true) jenkins-hbase9.apache.org,33419,1689325591459/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@66ac4a42[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-14 09:06:54,775 INFO [PEWorker-2] procedure.ServerCrashProcedure(161): Start pid=21, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,33419,1689325591459, splitWal=true, meta=true 2023-07-14 09:06:54,776 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=21, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,33419,1689325591459, splitWal=true, meta=true, isMeta: true 2023-07-14 09:06:54,780 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting 2023-07-14 09:06:54,781 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase9.apache.org,33419,1689325591459 WAL count=1, meta=true 2023-07-14 09:06:54,781 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta}] 2023-07-14 09:06:54,784 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:54,785 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta, worker=jenkins-hbase9.apache.org,34111,1689325597253}] 2023-07-14 09:06:54,867 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:33419 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:33419 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 09:06:54,868 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:33419 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:33419 2023-07-14 09:06:54,871 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:33419-0x101630797730002, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 09:06:54,871 INFO [RS:1;jenkins-hbase9:33419] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,33419,1689325591459; zookeeper connection closed. 2023-07-14 09:06:54,871 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:33419-0x101630797730002, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 09:06:54,871 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@35cd8c21] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@35cd8c21 2023-07-14 09:06:54,937 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:54,938 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-14 09:06:54,939 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:51656, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-14 09:06:54,939 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34111] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=23 2023-07-14 09:06:54,955 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta, size=1.3 K (1341bytes) 2023-07-14 09:06:54,955 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta 2023-07-14 09:06:54,956 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta after 0ms 2023-07-14 09:06:54,960 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(311): Open hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta took 6ms 2023-07-14 09:06:54,963 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 13 store_sequence_id { family_name: "info" sequence_id: 15 } store_sequence_id { family_name: "rep_barrier" sequence_id: 13 } store_sequence_id { family_name: "table" sequence_id: 13 } 2023-07-14 09:06:54,964 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta so closing down 2023-07-14 09:06:54,964 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-14 09:06:54,968 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta.temp 2023-07-14 09:06:54,968 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-14 09:06:54,978 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-14 09:06:54,979 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-14 09:06:54,981 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta.temp to hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-14 09:06:54,982 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(423): Processed 3 edits across 1 Regions in 22 ms; skipped=1; WAL=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta, size=1.3 K, length=1341, corrupted=false, cancelled=false 2023-07-14 09:06:54,982 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta, journal: Splitting hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta, size=1.3 K (1341bytes) at 1689325614955Finishing writing output for hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta so closing down at 1689325614964 (+9 ms)Creating recovered edits writer path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta.temp at 1689325614968 (+4 ms)3 split writer threads finished at 1689325614968Closed recovered edits writer path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689325614980 (+12 ms)Rename recovered edits hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta.temp to hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 at 1689325614982 (+2 ms)Processed 3 edits across 1 Regions in 22 ms; skipped=1; WAL=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta, size=1.3 K, length=1341, corrupted=false, cancelled=false at 1689325614982 2023-07-14 09:06:54,982 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta 2023-07-14 09:06:54,982 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=23 2023-07-14 09:06:54,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44817] master.HMaster(3641): Remote procedure done, pid=23 2023-07-14 09:06:54,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44817] wal.WALSplitUtil(141): Moved hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta to hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/oldWALs 2023-07-14 09:06:54,989 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-14 09:06:54,989 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta, worker=jenkins-hbase9.apache.org,34111,1689325597253 in 202 msec 2023-07-14 09:06:54,991 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:54,993 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-07-14 09:06:54,993 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; SplitWALProcedure jenkins-hbase9.apache.org%2C33419%2C1689325591459.meta.1689325595826.meta, worker=jenkins-hbase9.apache.org,34111,1689325597253 in 210 msec 2023-07-14 09:06:54,995 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting dir is empty, no logs to split. 2023-07-14 09:06:54,995 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase9.apache.org,33419,1689325591459 WAL count=0, meta=true 2023-07-14 09:06:54,995 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase9.apache.org,33419,1689325591459 WAL splitting is done? wals=0, meta=true 2023-07-14 09:06:54,996 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-14 09:06:54,997 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-14 09:06:54,998 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 09:06:55,148 DEBUG [jenkins-hbase9:44817] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-14 09:06:55,149 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,34111,1689325597253, state=OPENING 2023-07-14 09:06:55,155 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-14 09:06:55,155 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 09:06:55,155 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,34111,1689325597253}] 2023-07-14 09:06:55,169 WARN [ReadOnlyZKClient-127.0.0.1:52519@0x7a742b87] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-14 09:06:55,169 DEBUG [hconnection-0x5587207f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 09:06:55,172 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:51658, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 09:06:55,172 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34111] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 111 connection: 172.31.2.10:51658 deadline: 1689325675172, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:55,312 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-14 09:06:55,312 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 09:06:55,315 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C34111%2C1689325597253.meta, suffix=.meta, logDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,34111,1689325597253, archiveDir=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/oldWALs, maxLogs=32 2023-07-14 09:06:55,331 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33565,DS-46c52e53-4cf7-418f-9976-a339137f48a3,DISK] 2023-07-14 09:06:55,331 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41183,DS-06b9980b-5da4-4372-b119-090a73b88c97,DISK] 2023-07-14 09:06:55,333 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,34111,1689325597253/jenkins-hbase9.apache.org%2C34111%2C1689325597253.meta.1689325615315.meta 2023-07-14 09:06:55,334 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33565,DS-46c52e53-4cf7-418f-9976-a339137f48a3,DISK], DatanodeInfoWithStorage[127.0.0.1:41183,DS-06b9980b-5da4-4372-b119-090a73b88c97,DISK]] 2023-07-14 09:06:55,334 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-14 09:06:55,334 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-14 09:06:55,334 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-14 09:06:55,334 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-14 09:06:55,334 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-14 09:06:55,334 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 09:06:55,334 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-14 09:06:55,334 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-14 09:06:55,336 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-14 09:06:55,337 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/info 2023-07-14 09:06:55,337 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/info 2023-07-14 09:06:55,338 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-14 09:06:55,347 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/info/0fc741c44d8847e585b28c4f1f4ce995 2023-07-14 09:06:55,348 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:55,348 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-14 09:06:55,349 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/rep_barrier 2023-07-14 09:06:55,349 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/rep_barrier 2023-07-14 09:06:55,350 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-14 09:06:55,350 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:55,351 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-14 09:06:55,352 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/table 2023-07-14 09:06:55,352 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/table 2023-07-14 09:06:55,353 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-14 09:06:55,362 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/table/c97c898f88734748bec14934376d6f27 2023-07-14 09:06:55,362 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:55,363 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740 2023-07-14 09:06:55,365 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740 2023-07-14 09:06:55,366 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-14 09:06:55,371 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5464): Applied 8, skipped 0, firstSequenceIdInLog=16, maxSequenceIdInLog=17, path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-14 09:06:55,371 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=834 B heapSize=2.16 KB 2023-07-14 09:06:55,384 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=834 B at sequenceid=17 (bloomFilter=false), to=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/.tmp/info/35a4268d2f00436e9873d2fc9468edc8 2023-07-14 09:06:55,391 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/.tmp/info/35a4268d2f00436e9873d2fc9468edc8 as hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/info/35a4268d2f00436e9873d2fc9468edc8 2023-07-14 09:06:55,397 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/info/35a4268d2f00436e9873d2fc9468edc8, entries=8, sequenceid=17, filesize=5.5 K 2023-07-14 09:06:55,398 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~834 B/834, heapSize ~1.64 KB/1680, currentSize=0 B/0 for 1588230740 in 27ms, sequenceid=17, compaction requested=false; wal=null 2023-07-14 09:06:55,399 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-14 09:06:55,401 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-14 09:06:55,403 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-14 09:06:55,405 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=13 2023-07-14 09:06:55,406 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=18; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72466621, jitterRate=0.07983680069446564}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-14 09:06:55,406 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-14 09:06:55,407 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=25, masterSystemTime=1689325615308 2023-07-14 09:06:55,410 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-14 09:06:55,411 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-14 09:06:55,411 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,34111,1689325597253, state=OPEN 2023-07-14 09:06:55,413 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-14 09:06:55,413 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 09:06:55,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-14 09:06:55,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,34111,1689325597253 in 258 msec 2023-07-14 09:06:55,418 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=21 2023-07-14 09:06:55,418 INFO [PEWorker-3] procedure.ServerCrashProcedure(199): jenkins-hbase9.apache.org,33419,1689325591459 had 3 regions 2023-07-14 09:06:55,418 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 419 msec 2023-07-14 09:06:55,419 INFO [PEWorker-3] procedure.ServerCrashProcedure(300): Splitting WALs pid=21, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,33419,1689325591459, splitWal=true, meta=true, isMeta: false 2023-07-14 09:06:55,421 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase9.apache.org,33419,1689325591459 WAL count=1, meta=false 2023-07-14 09:06:55,421 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=21, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810}] 2023-07-14 09:06:55,423 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:55,424 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810, worker=jenkins-hbase9.apache.org,34111,1689325597253}] 2023-07-14 09:06:55,576 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34111] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=27 2023-07-14 09:06:55,590 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810, size=899 (899bytes) 2023-07-14 09:06:55,590 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810 2023-07-14 09:06:55,590 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810 after 0ms 2023-07-14 09:06:55,593 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(311): Open hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810 took 4ms 2023-07-14 09:06:55,597 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(346): Last flushed sequenceid for fc5008608be0e2d76472fa37e4d943d7: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-14 09:06:55,599 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(346): Last flushed sequenceid for b2ab292f4fc17b21d3c864d08ea57223: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-14 09:06:55,599 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810 so closing down 2023-07-14 09:06:55,600 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-14 09:06:55,602 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810.temp 2023-07-14 09:06:55,602 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-14 09:06:55,615 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-14 09:06:55,617 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810.temp to hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/recovered.edits/0000000000000000005 2023-07-14 09:06:55,617 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(423): Processed 4 edits across 1 Regions in 23 ms; skipped=2; WAL=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810, size=899, length=899, corrupted=false, cancelled=false 2023-07-14 09:06:55,617 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810, journal: Splitting hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810, size=899 (899bytes) at 1689325615590Finishing writing output for hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810 so closing down at 1689325615599 (+9 ms)Creating recovered edits writer path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810.temp at 1689325615602 (+3 ms)3 split writer threads finished at 1689325615602Closed recovered edits writer path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689325615615 (+13 ms)Rename recovered edits hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810.temp to hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/recovered.edits/0000000000000000005 at 1689325615617 (+2 ms)Processed 4 edits across 1 Regions in 23 ms; skipped=2; WAL=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810, size=899, length=899, corrupted=false, cancelled=false at 1689325615617 2023-07-14 09:06:55,617 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810 2023-07-14 09:06:55,617 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=27 2023-07-14 09:06:55,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44817] master.HMaster(3641): Remote procedure done, pid=27 2023-07-14 09:06:55,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44817] wal.WALSplitUtil(141): Moved hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting/jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810 to hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/oldWALs 2023-07-14 09:06:55,623 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=26 2023-07-14 09:06:55,624 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810, worker=jenkins-hbase9.apache.org,34111,1689325597253 in 197 msec 2023-07-14 09:06:55,625 DEBUG [PEWorker-2] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:55,627 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=21 2023-07-14 09:06:55,627 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=21, state=SUCCESS; SplitWALProcedure jenkins-hbase9.apache.org%2C33419%2C1689325591459.1689325592810, worker=jenkins-hbase9.apache.org,34111,1689325597253 in 204 msec 2023-07-14 09:06:55,629 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/WALs/jenkins-hbase9.apache.org,33419,1689325591459-splitting dir is empty, no logs to split. 2023-07-14 09:06:55,629 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase9.apache.org,33419,1689325591459 WAL count=0, meta=false 2023-07-14 09:06:55,629 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase9.apache.org,33419,1689325591459 WAL splitting is done? wals=0, meta=false 2023-07-14 09:06:55,631 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=b2ab292f4fc17b21d3c864d08ea57223, ASSIGN}, {pid=29, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=fc5008608be0e2d76472fa37e4d943d7, ASSIGN}] 2023-07-14 09:06:55,632 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=fc5008608be0e2d76472fa37e4d943d7, ASSIGN 2023-07-14 09:06:55,632 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=b2ab292f4fc17b21d3c864d08ea57223, ASSIGN 2023-07-14 09:06:55,633 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=fc5008608be0e2d76472fa37e4d943d7, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 09:06:55,633 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=28, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=b2ab292f4fc17b21d3c864d08ea57223, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 09:06:55,686 DEBUG [hconnection-0x5587207f-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase9.apache.org/172.31.2.10:33419 this server is in the failed servers list 2023-07-14 09:06:55,783 DEBUG [jenkins-hbase9:44817] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-14 09:06:55,784 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=b2ab292f4fc17b21d3c864d08ea57223, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:55,784 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=fc5008608be0e2d76472fa37e4d943d7, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:55,784 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689325615783"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689325615783"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689325615783"}]},"ts":"1689325615783"} 2023-07-14 09:06:55,784 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689325615783"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689325615783"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689325615783"}]},"ts":"1689325615783"} 2023-07-14 09:06:55,785 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:33419 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:33419 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 09:06:55,786 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:33419 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:33419 2023-07-14 09:06:55,794 DEBUG [hconnection-0x5587207f-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase9.apache.org/172.31.2.10:33419 this server is in the failed servers list 2023-07-14 09:06:55,887 DEBUG [PEWorker-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 09:06:55,889 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:51670, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 09:06:55,891 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure fc5008608be0e2d76472fa37e4d943d7, server=jenkins-hbase9.apache.org,34111,1689325597253}] 2023-07-14 09:06:55,892 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=28, state=RUNNABLE; OpenRegionProcedure b2ab292f4fc17b21d3c864d08ea57223, server=jenkins-hbase9.apache.org,34111,1689325597253}] 2023-07-14 09:06:56,000 DEBUG [hconnection-0x5587207f-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase9.apache.org/172.31.2.10:33419 this server is in the failed servers list 2023-07-14 09:06:56,047 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:56,048 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fc5008608be0e2d76472fa37e4d943d7, NAME => 'hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7.', STARTKEY => '', ENDKEY => ''} 2023-07-14 09:06:56,048 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:56,048 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 09:06:56,048 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:56,048 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:56,050 INFO [StoreOpener-fc5008608be0e2d76472fa37e4d943d7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:56,050 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:56,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b2ab292f4fc17b21d3c864d08ea57223, NAME => 't,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.', STARTKEY => '', ENDKEY => ''} 2023-07-14 09:06:56,051 DEBUG [StoreOpener-fc5008608be0e2d76472fa37e4d943d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/info 2023-07-14 09:06:56,051 DEBUG [StoreOpener-fc5008608be0e2d76472fa37e4d943d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/info 2023-07-14 09:06:56,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:56,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 09:06:56,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:56,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:56,051 INFO [StoreOpener-fc5008608be0e2d76472fa37e4d943d7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fc5008608be0e2d76472fa37e4d943d7 columnFamilyName info 2023-07-14 09:06:56,052 INFO [StoreOpener-fc5008608be0e2d76472fa37e4d943d7-1] regionserver.HStore(310): Store=fc5008608be0e2d76472fa37e4d943d7/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:56,052 INFO [StoreOpener-b2ab292f4fc17b21d3c864d08ea57223-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:56,054 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:56,054 DEBUG [StoreOpener-b2ab292f4fc17b21d3c864d08ea57223-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/f 2023-07-14 09:06:56,054 DEBUG [StoreOpener-b2ab292f4fc17b21d3c864d08ea57223-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/f 2023-07-14 09:06:56,055 INFO [StoreOpener-b2ab292f4fc17b21d3c864d08ea57223-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b2ab292f4fc17b21d3c864d08ea57223 columnFamilyName f 2023-07-14 09:06:56,056 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:56,056 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/recovered.edits/0000000000000000005 2023-07-14 09:06:56,060 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/recovered.edits/0000000000000000005 2023-07-14 09:06:56,060 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing fc5008608be0e2d76472fa37e4d943d7 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-14 09:06:56,062 DEBUG [StoreOpener-b2ab292f4fc17b21d3c864d08ea57223-1] regionserver.HStore(539): loaded hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/f/41bb6c12e40f43ad912f1d7c2fff5c75 2023-07-14 09:06:56,063 INFO [StoreOpener-b2ab292f4fc17b21d3c864d08ea57223-1] regionserver.HStore(310): Store=b2ab292f4fc17b21d3c864d08ea57223/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 09:06:56,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:56,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:56,069 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:56,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-14 09:06:56,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened b2ab292f4fc17b21d3c864d08ea57223; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73214167, jitterRate=0.09097610414028168}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 09:06:56,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for b2ab292f4fc17b21d3c864d08ea57223: 2023-07-14 09:06:56,075 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223., pid=31, masterSystemTime=1689325616043 2023-07-14 09:06:56,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:56,077 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:56,079 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=b2ab292f4fc17b21d3c864d08ea57223, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:56,082 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689325616079"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689325616079"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689325616079"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689325616079"}]},"ts":"1689325616079"} 2023-07-14 09:06:56,085 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/.tmp/info/e22ce3f779ec499ba214da8e9b0ca7ac 2023-07-14 09:06:56,088 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=28 2023-07-14 09:06:56,088 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=28, state=SUCCESS; OpenRegionProcedure b2ab292f4fc17b21d3c864d08ea57223, server=jenkins-hbase9.apache.org,34111,1689325597253 in 194 msec 2023-07-14 09:06:56,089 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=t, region=b2ab292f4fc17b21d3c864d08ea57223, ASSIGN in 457 msec 2023-07-14 09:06:56,093 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/.tmp/info/e22ce3f779ec499ba214da8e9b0ca7ac as hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/info/e22ce3f779ec499ba214da8e9b0ca7ac 2023-07-14 09:06:56,100 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/info/e22ce3f779ec499ba214da8e9b0ca7ac, entries=2, sequenceid=5, filesize=4.8 K 2023-07-14 09:06:56,100 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for fc5008608be0e2d76472fa37e4d943d7 in 40ms, sequenceid=5, compaction requested=false; wal=null 2023-07-14 09:06:56,101 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/recovered.edits/0000000000000000005 2023-07-14 09:06:56,105 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:56,108 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-14 09:06:56,109 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened fc5008608be0e2d76472fa37e4d943d7; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60082647, jitterRate=-0.1046987920999527}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 09:06:56,109 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for fc5008608be0e2d76472fa37e4d943d7: 2023-07-14 09:06:56,110 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7., pid=30, masterSystemTime=1689325616043 2023-07-14 09:06:56,112 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:56,112 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:56,113 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=fc5008608be0e2d76472fa37e4d943d7, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:56,113 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689325616113"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689325616113"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689325616113"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689325616113"}]},"ts":"1689325616113"} 2023-07-14 09:06:56,117 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-07-14 09:06:56,117 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure fc5008608be0e2d76472fa37e4d943d7, server=jenkins-hbase9.apache.org,34111,1689325597253 in 224 msec 2023-07-14 09:06:56,119 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=21 2023-07-14 09:06:56,120 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase9.apache.org,33419,1689325591459 after splitting done 2023-07-14 09:06:56,120 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=fc5008608be0e2d76472fa37e4d943d7, ASSIGN in 486 msec 2023-07-14 09:06:56,120 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase9.apache.org,33419,1689325591459 from processing; numProcessing=0 2023-07-14 09:06:56,122 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, state=SUCCESS; ServerCrashProcedure jenkins-hbase9.apache.org,33419,1689325591459, splitWal=true, meta=true in 1.3480 sec 2023-07-14 09:06:56,311 DEBUG [Listener at localhost/45201] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a742b87 to 127.0.0.1:52519 2023-07-14 09:06:56,311 DEBUG [Listener at localhost/45201] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 09:06:56,311 INFO [Listener at localhost/45201] client.TestScannerTimeout(124): END ************ test2772 2023-07-14 09:06:56,325 INFO [Listener at localhost/45201] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=275 (was 294), OpenFileDescriptor=583 (was 573) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=118 (was 118), ProcessCount=174 (was 174), AvailableMemoryMB=10167 (was 10183) 2023-07-14 09:06:56,325 INFO [Listener at localhost/45201] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-14 09:06:56,325 INFO [Listener at localhost/45201] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-14 09:06:56,325 DEBUG [Listener at localhost/45201] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x511bead0 to 127.0.0.1:52519 2023-07-14 09:06:56,325 DEBUG [Listener at localhost/45201] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 09:06:56,325 DEBUG [Listener at localhost/45201] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-14 09:06:56,326 DEBUG [Listener at localhost/45201] util.JVMClusterUtil(257): Found active master hash=1329579802, stopped=false 2023-07-14 09:06:56,326 INFO [Listener at localhost/45201] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase9.apache.org,44817,1689325590279 2023-07-14 09:06:56,328 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:34111-0x101630797730008, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-14 09:06:56,328 INFO [Listener at localhost/45201] procedure2.ProcedureExecutor(629): Stopping 2023-07-14 09:06:56,328 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-14 09:06:56,328 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 09:06:56,328 DEBUG [Listener at localhost/45201] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52be7963 to 127.0.0.1:52519 2023-07-14 09:06:56,329 DEBUG [Listener at localhost/45201] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 09:06:56,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34111-0x101630797730008, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 09:06:56,329 INFO [Listener at localhost/45201] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,34111,1689325597253' ***** 2023-07-14 09:06:56,329 INFO [Listener at localhost/45201] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-14 09:06:56,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 09:06:56,329 INFO [RS:2;jenkins-hbase9:34111] regionserver.HeapMemoryManager(220): Stopping 2023-07-14 09:06:56,329 INFO [RS:2;jenkins-hbase9:34111] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-14 09:06:56,329 INFO [RS:2;jenkins-hbase9:34111] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-14 09:06:56,330 INFO [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(3305): Received CLOSE for b2ab292f4fc17b21d3c864d08ea57223 2023-07-14 09:06:56,330 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-14 09:06:56,330 INFO [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(3305): Received CLOSE for fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:56,330 INFO [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(1144): stopping server jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:56,330 DEBUG [RS:2;jenkins-hbase9:34111] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x407d9f81 to 127.0.0.1:52519 2023-07-14 09:06:56,330 DEBUG [RS:2;jenkins-hbase9:34111] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 09:06:56,330 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing b2ab292f4fc17b21d3c864d08ea57223, disabling compactions & flushes 2023-07-14 09:06:56,330 INFO [RS:2;jenkins-hbase9:34111] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-14 09:06:56,331 INFO [RS:2;jenkins-hbase9:34111] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-14 09:06:56,331 INFO [RS:2;jenkins-hbase9:34111] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-14 09:06:56,330 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:56,331 INFO [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-14 09:06:56,331 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:56,331 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. after waiting 0 ms 2023-07-14 09:06:56,331 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:56,331 INFO [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-14 09:06:56,331 DEBUG [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, b2ab292f4fc17b21d3c864d08ea57223=t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223., fc5008608be0e2d76472fa37e4d943d7=hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7.} 2023-07-14 09:06:56,331 DEBUG [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(1504): Waiting on 1588230740, b2ab292f4fc17b21d3c864d08ea57223, fc5008608be0e2d76472fa37e4d943d7 2023-07-14 09:06:56,331 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-14 09:06:56,332 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-14 09:06:56,332 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-14 09:06:56,332 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-14 09:06:56,332 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-14 09:06:56,332 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-14 09:06:56,337 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/default/t/b2ab292f4fc17b21d3c864d08ea57223/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-14 09:06:56,338 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:56,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for b2ab292f4fc17b21d3c864d08ea57223: 2023-07-14 09:06:56,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed t,,1689325594152.b2ab292f4fc17b21d3c864d08ea57223. 2023-07-14 09:06:56,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing fc5008608be0e2d76472fa37e4d943d7, disabling compactions & flushes 2023-07-14 09:06:56,339 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:56,339 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:56,339 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. after waiting 0 ms 2023-07-14 09:06:56,339 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:56,344 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/namespace/fc5008608be0e2d76472fa37e4d943d7/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=5 2023-07-14 09:06:56,345 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:56,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for fc5008608be0e2d76472fa37e4d943d7: 2023-07-14 09:06:56,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689325593317.fc5008608be0e2d76472fa37e4d943d7. 2023-07-14 09:06:56,355 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/.tmp/info/6a9fef7ec3174f62b88d48ae98a43a69 2023-07-14 09:06:56,362 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/.tmp/info/6a9fef7ec3174f62b88d48ae98a43a69 as hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/info/6a9fef7ec3174f62b88d48ae98a43a69 2023-07-14 09:06:56,368 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/info/6a9fef7ec3174f62b88d48ae98a43a69, entries=16, sequenceid=24, filesize=6.4 K 2023-07-14 09:06:56,369 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 37ms, sequenceid=24, compaction requested=true 2023-07-14 09:06:56,375 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=17 2023-07-14 09:06:56,376 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-14 09:06:56,376 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-14 09:06:56,376 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-14 09:06:56,376 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-14 09:06:56,400 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-14 09:06:56,400 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-14 09:06:56,531 INFO [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,34111,1689325597253; all regions closed. 2023-07-14 09:06:56,537 DEBUG [RS:2;jenkins-hbase9:34111] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/oldWALs 2023-07-14 09:06:56,537 INFO [RS:2;jenkins-hbase9:34111] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C34111%2C1689325597253.meta:.meta(num 1689325615315) 2023-07-14 09:06:56,542 DEBUG [RS:2;jenkins-hbase9:34111] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/oldWALs 2023-07-14 09:06:56,542 INFO [RS:2;jenkins-hbase9:34111] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C34111%2C1689325597253:(num 1689325597457) 2023-07-14 09:06:56,542 DEBUG [RS:2;jenkins-hbase9:34111] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 09:06:56,542 INFO [RS:2;jenkins-hbase9:34111] regionserver.LeaseManager(133): Closed leases 2023-07-14 09:06:56,543 INFO [RS:2;jenkins-hbase9:34111] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-14 09:06:56,543 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-14 09:06:56,543 INFO [RS:2;jenkins-hbase9:34111] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:34111 2023-07-14 09:06:56,546 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 09:06:56,546 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:34111-0x101630797730008, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,34111,1689325597253 2023-07-14 09:06:56,546 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:34111-0x101630797730008, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 09:06:56,548 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,34111,1689325597253] 2023-07-14 09:06:56,548 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,34111,1689325597253; numProcessing=1 2023-07-14 09:06:56,549 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,34111,1689325597253 already deleted, retry=false 2023-07-14 09:06:56,549 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,34111,1689325597253 expired; onlineServers=0 2023-07-14 09:06:56,549 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,44817,1689325590279' ***** 2023-07-14 09:06:56,549 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-14 09:06:56,550 DEBUG [M:0;jenkins-hbase9:44817] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e02cd21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-14 09:06:56,550 INFO [M:0;jenkins-hbase9:44817] regionserver.HRegionServer(1144): stopping server jenkins-hbase9.apache.org,44817,1689325590279 2023-07-14 09:06:56,550 INFO [M:0;jenkins-hbase9:44817] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,44817,1689325590279; all regions closed. 2023-07-14 09:06:56,550 DEBUG [M:0;jenkins-hbase9:44817] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 09:06:56,550 DEBUG [M:0;jenkins-hbase9:44817] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-14 09:06:56,550 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-14 09:06:56,550 DEBUG [M:0;jenkins-hbase9:44817] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-14 09:06:56,550 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689325592517] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689325592517,5,FailOnTimeoutGroup] 2023-07-14 09:06:56,550 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689325592517] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689325592517,5,FailOnTimeoutGroup] 2023-07-14 09:06:56,550 INFO [M:0;jenkins-hbase9:44817] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-14 09:06:56,551 INFO [M:0;jenkins-hbase9:44817] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-14 09:06:56,551 INFO [M:0;jenkins-hbase9:44817] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-07-14 09:06:56,551 DEBUG [M:0;jenkins-hbase9:44817] master.HMaster(1512): Stopping service threads 2023-07-14 09:06:56,551 INFO [M:0;jenkins-hbase9:44817] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-14 09:06:56,551 ERROR [M:0;jenkins-hbase9:44817] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (494579167) connection to localhost/127.0.0.1:46473 from jenkins,5,PEWorkerGroup] 2023-07-14 09:06:56,551 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-14 09:06:56,551 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 09:06:56,552 INFO [M:0;jenkins-hbase9:44817] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-14 09:06:56,552 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-14 09:06:56,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-14 09:06:56,552 DEBUG [M:0;jenkins-hbase9:44817] zookeeper.ZKUtil(398): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-14 09:06:56,552 WARN [M:0;jenkins-hbase9:44817] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-14 09:06:56,552 INFO [M:0;jenkins-hbase9:44817] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-14 09:06:56,552 INFO [M:0;jenkins-hbase9:44817] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-14 09:06:56,553 DEBUG [M:0;jenkins-hbase9:44817] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-14 09:06:56,553 INFO [M:0;jenkins-hbase9:44817] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 09:06:56,553 DEBUG [M:0;jenkins-hbase9:44817] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 09:06:56,553 DEBUG [M:0;jenkins-hbase9:44817] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-14 09:06:56,553 DEBUG [M:0;jenkins-hbase9:44817] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 09:06:56,553 INFO [M:0;jenkins-hbase9:44817] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=105.75 KB heapSize=127.45 KB 2023-07-14 09:06:56,567 INFO [M:0;jenkins-hbase9:44817] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=105.75 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/311ab29919794c8f9de8bf26f924640c 2023-07-14 09:06:56,573 DEBUG [M:0;jenkins-hbase9:44817] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/311ab29919794c8f9de8bf26f924640c as hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/311ab29919794c8f9de8bf26f924640c 2023-07-14 09:06:56,579 INFO [M:0;jenkins-hbase9:44817] regionserver.HStore(1080): Added hdfs://localhost:46473/user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/311ab29919794c8f9de8bf26f924640c, entries=31, sequenceid=244, filesize=8.4 K 2023-07-14 09:06:56,580 INFO [M:0;jenkins-hbase9:44817] regionserver.HRegion(2948): Finished flush of dataSize ~105.75 KB/108290, heapSize ~127.44 KB/130496, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 27ms, sequenceid=244, compaction requested=false 2023-07-14 09:06:56,581 INFO [M:0;jenkins-hbase9:44817] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 09:06:56,581 DEBUG [M:0;jenkins-hbase9:44817] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-14 09:06:56,584 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/7150740d-92f7-c08f-041a-4f2bf7b0dfd4/MasterData/WALs/jenkins-hbase9.apache.org,44817,1689325590279/jenkins-hbase9.apache.org%2C44817%2C1689325590279.1689325591898 not finished, retry = 0 2023-07-14 09:06:56,685 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-14 09:06:56,685 INFO [M:0;jenkins-hbase9:44817] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-14 09:06:56,686 INFO [M:0;jenkins-hbase9:44817] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:44817 2023-07-14 09:06:56,689 DEBUG [M:0;jenkins-hbase9:44817] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,44817,1689325590279 already deleted, retry=false 2023-07-14 09:06:56,815 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 09:06:56,815 INFO [M:0;jenkins-hbase9:44817] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,44817,1689325590279; zookeeper connection closed. 2023-07-14 09:06:56,815 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): master:44817-0x101630797730000, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 09:06:56,915 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:34111-0x101630797730008, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 09:06:56,915 INFO [RS:2;jenkins-hbase9:34111] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,34111,1689325597253; zookeeper connection closed. 2023-07-14 09:06:56,915 DEBUG [Listener at localhost/45201-EventThread] zookeeper.ZKWatcher(600): regionserver:34111-0x101630797730008, quorum=127.0.0.1:52519, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 09:06:56,916 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7e077ea1] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7e077ea1 2023-07-14 09:06:56,916 INFO [Listener at localhost/45201] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-14 09:06:56,916 WARN [Listener at localhost/45201] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-14 09:06:56,919 INFO [Listener at localhost/45201] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-14 09:06:57,024 WARN [BP-514055984-172.31.2.10-1689325587359 heartbeating to localhost/127.0.0.1:46473] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-14 09:06:57,024 WARN [BP-514055984-172.31.2.10-1689325587359 heartbeating to localhost/127.0.0.1:46473] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-514055984-172.31.2.10-1689325587359 (Datanode Uuid 061ef11e-14fd-4f2e-801d-3d083e37c651) service to localhost/127.0.0.1:46473 2023-07-14 09:06:57,026 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/cluster_708bca9d-6cdf-4da8-0283-1300549de5a3/dfs/data/data3/current/BP-514055984-172.31.2.10-1689325587359] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-14 09:06:57,026 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/cluster_708bca9d-6cdf-4da8-0283-1300549de5a3/dfs/data/data4/current/BP-514055984-172.31.2.10-1689325587359] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-14 09:06:57,028 WARN [Listener at localhost/45201] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-14 09:06:57,029 INFO [Listener at localhost/45201] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-14 09:06:57,134 WARN [BP-514055984-172.31.2.10-1689325587359 heartbeating to localhost/127.0.0.1:46473] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-14 09:06:57,134 WARN [BP-514055984-172.31.2.10-1689325587359 heartbeating to localhost/127.0.0.1:46473] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-514055984-172.31.2.10-1689325587359 (Datanode Uuid 250924d0-68c2-457f-b965-8dbeecbad572) service to localhost/127.0.0.1:46473 2023-07-14 09:06:57,135 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/cluster_708bca9d-6cdf-4da8-0283-1300549de5a3/dfs/data/data1/current/BP-514055984-172.31.2.10-1689325587359] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-14 09:06:57,135 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/95746da0-64e6-03c3-26e6-3ec8149a7408/cluster_708bca9d-6cdf-4da8-0283-1300549de5a3/dfs/data/data2/current/BP-514055984-172.31.2.10-1689325587359] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-14 09:06:57,170 INFO [Listener at localhost/45201] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-14 09:06:57,282 INFO [Listener at localhost/45201] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-14 09:06:57,316 INFO [Listener at localhost/45201] hbase.HBaseTestingUtility(1293): Minicluster is down