2023-07-14 13:06:24,723 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-14 13:06:24,940 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d 2023-07-14 13:06:24,954 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-14 13:06:24,955 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/cluster_b78f3397-94fd-c6dd-412f-27eb5739866b, deleteOnExit=true 2023-07-14 13:06:24,955 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-14 13:06:24,956 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/test.cache.data in system properties and HBase conf 2023-07-14 13:06:24,956 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/hadoop.tmp.dir in system properties and HBase conf 2023-07-14 13:06:24,957 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/hadoop.log.dir in system properties and HBase conf 2023-07-14 13:06:24,957 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-14 13:06:24,958 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-14 13:06:24,958 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-14 13:06:25,072 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-14 13:06:25,501 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-14 13:06:25,504 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-14 13:06:25,504 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-14 13:06:25,505 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-14 13:06:25,505 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-14 13:06:25,506 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-14 13:06:25,506 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-14 13:06:25,507 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-14 13:06:25,507 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-14 13:06:25,507 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-14 13:06:25,508 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/nfs.dump.dir in system properties and HBase conf 2023-07-14 13:06:25,508 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/java.io.tmpdir in system properties and HBase conf 2023-07-14 13:06:25,509 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-14 13:06:25,509 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-14 13:06:25,510 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-14 13:06:26,012 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-14 13:06:26,017 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-14 13:06:26,353 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-14 13:06:26,551 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-14 13:06:26,565 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-14 13:06:26,599 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-14 13:06:26,629 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/java.io.tmpdir/Jetty_localhost_35029_hdfs____.2crfcx/webapp 2023-07-14 13:06:26,751 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35029 2023-07-14 13:06:26,768 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-14 13:06:26,769 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-14 13:06:27,236 WARN [Listener at localhost/34955] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-14 13:06:27,322 WARN [Listener at localhost/34955] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-14 13:06:27,339 WARN [Listener at localhost/34955] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-14 13:06:27,344 INFO [Listener at localhost/34955] log.Slf4jLog(67): jetty-6.1.26 2023-07-14 13:06:27,348 INFO [Listener at localhost/34955] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/java.io.tmpdir/Jetty_localhost_46599_datanode____.vs2ibq/webapp 2023-07-14 13:06:27,441 INFO [Listener at localhost/34955] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46599 2023-07-14 13:06:27,750 WARN [Listener at localhost/40061] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-14 13:06:27,765 WARN [Listener at localhost/40061] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-14 13:06:27,769 WARN [Listener at localhost/40061] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-14 13:06:27,772 INFO [Listener at localhost/40061] log.Slf4jLog(67): jetty-6.1.26 2023-07-14 13:06:27,778 INFO [Listener at localhost/40061] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/java.io.tmpdir/Jetty_localhost_46415_datanode____tl30cd/webapp 2023-07-14 13:06:27,888 INFO [Listener at localhost/40061] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46415 2023-07-14 13:06:27,901 WARN [Listener at localhost/40033] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-14 13:06:28,202 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x6ff7761134354056: Processing first storage report for DS-d0e9d4fb-a902-4f72-9d50-25e21bea1880 from datanode 3db6d24c-9770-4b94-bf1d-ce8bea397f3a 2023-07-14 13:06:28,203 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x6ff7761134354056: from storage DS-d0e9d4fb-a902-4f72-9d50-25e21bea1880 node DatanodeRegistration(127.0.0.1:45715, datanodeUuid=3db6d24c-9770-4b94-bf1d-ce8bea397f3a, infoPort=41555, infoSecurePort=0, ipcPort=40033, storageInfo=lv=-57;cid=testClusterID;nsid=1351863548;c=1689339986089), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-14 13:06:28,203 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8549a1d3d250ef74: Processing first storage report for DS-d1fe6f1d-a2c6-4e10-b2ab-1a91df578e1c from datanode 6cba91a1-18bb-431c-94a7-1b250f37eb9d 2023-07-14 13:06:28,204 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8549a1d3d250ef74: from storage DS-d1fe6f1d-a2c6-4e10-b2ab-1a91df578e1c node DatanodeRegistration(127.0.0.1:44567, datanodeUuid=6cba91a1-18bb-431c-94a7-1b250f37eb9d, infoPort=38217, infoSecurePort=0, ipcPort=40061, storageInfo=lv=-57;cid=testClusterID;nsid=1351863548;c=1689339986089), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-14 13:06:28,204 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x6ff7761134354056: Processing first storage report for DS-53d2b6b1-c5cf-4b31-903e-59dab4713f29 from datanode 3db6d24c-9770-4b94-bf1d-ce8bea397f3a 2023-07-14 13:06:28,204 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x6ff7761134354056: from storage DS-53d2b6b1-c5cf-4b31-903e-59dab4713f29 node DatanodeRegistration(127.0.0.1:45715, datanodeUuid=3db6d24c-9770-4b94-bf1d-ce8bea397f3a, infoPort=41555, infoSecurePort=0, ipcPort=40033, storageInfo=lv=-57;cid=testClusterID;nsid=1351863548;c=1689339986089), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-14 13:06:28,204 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8549a1d3d250ef74: Processing first storage report for DS-5b2ae68d-901a-44ce-bca6-fd72c33300a7 from datanode 6cba91a1-18bb-431c-94a7-1b250f37eb9d 2023-07-14 13:06:28,204 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8549a1d3d250ef74: from storage DS-5b2ae68d-901a-44ce-bca6-fd72c33300a7 node DatanodeRegistration(127.0.0.1:44567, datanodeUuid=6cba91a1-18bb-431c-94a7-1b250f37eb9d, infoPort=38217, infoSecurePort=0, ipcPort=40061, storageInfo=lv=-57;cid=testClusterID;nsid=1351863548;c=1689339986089), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-14 13:06:28,284 DEBUG [Listener at localhost/40033] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d 2023-07-14 13:06:28,349 INFO [Listener at localhost/40033] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/cluster_b78f3397-94fd-c6dd-412f-27eb5739866b/zookeeper_0, clientPort=60854, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/cluster_b78f3397-94fd-c6dd-412f-27eb5739866b/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/cluster_b78f3397-94fd-c6dd-412f-27eb5739866b/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-14 13:06:28,362 INFO [Listener at localhost/40033] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=60854 2023-07-14 13:06:28,374 INFO [Listener at localhost/40033] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 13:06:28,376 INFO [Listener at localhost/40033] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 13:06:29,038 INFO [Listener at localhost/40033] util.FSUtils(471): Created version file at hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf with version=8 2023-07-14 13:06:29,038 INFO [Listener at localhost/40033] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/hbase-staging 2023-07-14 13:06:29,346 INFO [Listener at localhost/40033] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-14 13:06:29,812 INFO [Listener at localhost/40033] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-07-14 13:06:29,842 INFO [Listener at localhost/40033] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 13:06:29,843 INFO [Listener at localhost/40033] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-14 13:06:29,843 INFO [Listener at localhost/40033] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-14 13:06:29,843 INFO [Listener at localhost/40033] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 13:06:29,843 INFO [Listener at localhost/40033] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-14 13:06:29,980 INFO [Listener at localhost/40033] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-14 13:06:30,054 DEBUG [Listener at localhost/40033] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-14 13:06:30,148 INFO [Listener at localhost/40033] ipc.NettyRpcServer(120): Bind to /172.31.2.10:45203 2023-07-14 13:06:30,158 INFO [Listener at localhost/40033] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 13:06:30,161 INFO [Listener at localhost/40033] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 13:06:30,181 INFO [Listener at localhost/40033] zookeeper.RecoverableZooKeeper(93): Process identifier=master:45203 connecting to ZooKeeper ensemble=127.0.0.1:60854 2023-07-14 13:06:30,220 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:452030x0, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-14 13:06:30,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:45203-0x10163e34d0a0000 connected 2023-07-14 13:06:30,246 DEBUG [Listener at localhost/40033] zookeeper.ZKUtil(164): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-14 13:06:30,247 DEBUG [Listener at localhost/40033] zookeeper.ZKUtil(164): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 13:06:30,250 DEBUG [Listener at localhost/40033] zookeeper.ZKUtil(164): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-14 13:06:30,257 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45203 2023-07-14 13:06:30,257 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45203 2023-07-14 13:06:30,258 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45203 2023-07-14 13:06:30,258 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45203 2023-07-14 13:06:30,259 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45203 2023-07-14 13:06:30,264 INFO [Listener at localhost/40033] master.HMaster(444): hbase.rootdir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf, hbase.cluster.distributed=false 2023-07-14 13:06:30,332 INFO [Listener at localhost/40033] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-07-14 13:06:30,332 INFO [Listener at localhost/40033] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 13:06:30,333 INFO [Listener at localhost/40033] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-14 13:06:30,333 INFO [Listener at localhost/40033] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-14 13:06:30,333 INFO [Listener at localhost/40033] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 13:06:30,333 INFO [Listener at localhost/40033] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-14 13:06:30,338 INFO [Listener at localhost/40033] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-14 13:06:30,341 INFO [Listener at localhost/40033] ipc.NettyRpcServer(120): Bind to /172.31.2.10:33743 2023-07-14 13:06:30,343 INFO [Listener at localhost/40033] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-14 13:06:30,349 DEBUG [Listener at localhost/40033] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-14 13:06:30,350 INFO [Listener at localhost/40033] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 13:06:30,352 INFO [Listener at localhost/40033] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 13:06:30,354 INFO [Listener at localhost/40033] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33743 connecting to ZooKeeper ensemble=127.0.0.1:60854 2023-07-14 13:06:30,357 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:337430x0, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-14 13:06:30,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:33743-0x10163e34d0a0001 connected 2023-07-14 13:06:30,358 DEBUG [Listener at localhost/40033] zookeeper.ZKUtil(164): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-14 13:06:30,359 DEBUG [Listener at localhost/40033] zookeeper.ZKUtil(164): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 13:06:30,360 DEBUG [Listener at localhost/40033] zookeeper.ZKUtil(164): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-14 13:06:30,361 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33743 2023-07-14 13:06:30,361 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33743 2023-07-14 13:06:30,361 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33743 2023-07-14 13:06:30,362 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33743 2023-07-14 13:06:30,363 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33743 2023-07-14 13:06:30,375 INFO [Listener at localhost/40033] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-07-14 13:06:30,375 INFO [Listener at localhost/40033] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 13:06:30,375 INFO [Listener at localhost/40033] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-14 13:06:30,375 INFO [Listener at localhost/40033] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-14 13:06:30,375 INFO [Listener at localhost/40033] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 13:06:30,376 INFO [Listener at localhost/40033] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-14 13:06:30,376 INFO [Listener at localhost/40033] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-14 13:06:30,377 INFO [Listener at localhost/40033] ipc.NettyRpcServer(120): Bind to /172.31.2.10:38891 2023-07-14 13:06:30,377 INFO [Listener at localhost/40033] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-14 13:06:30,379 DEBUG [Listener at localhost/40033] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-14 13:06:30,380 INFO [Listener at localhost/40033] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 13:06:30,382 INFO [Listener at localhost/40033] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 13:06:30,384 INFO [Listener at localhost/40033] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38891 connecting to ZooKeeper ensemble=127.0.0.1:60854 2023-07-14 13:06:30,386 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:388910x0, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-14 13:06:30,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:38891-0x10163e34d0a0002 connected 2023-07-14 13:06:30,387 DEBUG [Listener at localhost/40033] zookeeper.ZKUtil(164): regionserver:38891-0x10163e34d0a0002, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-14 13:06:30,388 DEBUG [Listener at localhost/40033] zookeeper.ZKUtil(164): regionserver:38891-0x10163e34d0a0002, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 13:06:30,389 DEBUG [Listener at localhost/40033] zookeeper.ZKUtil(164): regionserver:38891-0x10163e34d0a0002, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-14 13:06:30,389 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38891 2023-07-14 13:06:30,390 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38891 2023-07-14 13:06:30,390 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38891 2023-07-14 13:06:30,390 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38891 2023-07-14 13:06:30,391 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38891 2023-07-14 13:06:30,392 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,45203,1689339989183 2023-07-14 13:06:30,401 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-14 13:06:30,402 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,45203,1689339989183 2023-07-14 13:06:30,420 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-14 13:06:30,420 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:38891-0x10163e34d0a0002, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-14 13:06:30,420 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-14 13:06:30,420 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 13:06:30,421 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-14 13:06:30,423 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,45203,1689339989183 from backup master directory 2023-07-14 13:06:30,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-14 13:06:30,425 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,45203,1689339989183 2023-07-14 13:06:30,426 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-14 13:06:30,426 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-14 13:06:30,426 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase9.apache.org,45203,1689339989183 2023-07-14 13:06:30,429 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-14 13:06:30,430 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-14 13:06:30,508 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/hbase.id with ID: 88dab552-822c-46d4-9648-1529c38da56c 2023-07-14 13:06:30,547 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 13:06:30,561 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 13:06:30,599 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2159cf25 to 127.0.0.1:60854 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 13:06:30,631 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ee66191, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 13:06:30,652 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-14 13:06:30,654 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-14 13:06:30,670 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-14 13:06:30,670 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-14 13:06:30,672 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-14 13:06:30,675 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-14 13:06:30,676 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 13:06:30,710 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/MasterData/data/master/store-tmp 2023-07-14 13:06:30,742 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 13:06:30,742 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-14 13:06:30,743 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 13:06:30,743 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 13:06:30,743 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-14 13:06:30,743 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 13:06:30,743 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 13:06:30,743 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-14 13:06:30,744 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/MasterData/WALs/jenkins-hbase9.apache.org,45203,1689339989183 2023-07-14 13:06:30,763 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C45203%2C1689339989183, suffix=, logDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/MasterData/WALs/jenkins-hbase9.apache.org,45203,1689339989183, archiveDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/MasterData/oldWALs, maxLogs=10 2023-07-14 13:06:30,826 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45715,DS-d0e9d4fb-a902-4f72-9d50-25e21bea1880,DISK] 2023-07-14 13:06:30,826 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44567,DS-d1fe6f1d-a2c6-4e10-b2ab-1a91df578e1c,DISK] 2023-07-14 13:06:30,834 DEBUG [RS-EventLoopGroup-4-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 13:06:30,894 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/MasterData/WALs/jenkins-hbase9.apache.org,45203,1689339989183/jenkins-hbase9.apache.org%2C45203%2C1689339989183.1689339990773 2023-07-14 13:06:30,894 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44567,DS-d1fe6f1d-a2c6-4e10-b2ab-1a91df578e1c,DISK], DatanodeInfoWithStorage[127.0.0.1:45715,DS-d0e9d4fb-a902-4f72-9d50-25e21bea1880,DISK]] 2023-07-14 13:06:30,895 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-14 13:06:30,895 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 13:06:30,898 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-14 13:06:30,899 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-14 13:06:30,952 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-14 13:06:30,958 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-14 13:06:30,981 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-14 13:06:30,992 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 13:06:30,997 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-14 13:06:30,998 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-14 13:06:31,013 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-14 13:06:31,018 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-14 13:06:31,019 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59246540, jitterRate=-0.11715775728225708}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 13:06:31,019 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-14 13:06:31,020 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-14 13:06:31,039 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-14 13:06:31,039 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-14 13:06:31,041 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-14 13:06:31,042 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-14 13:06:31,073 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 30 msec 2023-07-14 13:06:31,073 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-14 13:06:31,097 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-14 13:06:31,103 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-14 13:06:31,126 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-14 13:06:31,129 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-14 13:06:31,130 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-14 13:06:31,135 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-14 13:06:31,138 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-14 13:06:31,141 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 13:06:31,142 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-14 13:06:31,143 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-14 13:06:31,154 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-14 13:06:31,159 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-14 13:06:31,159 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-14 13:06:31,159 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:38891-0x10163e34d0a0002, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-14 13:06:31,159 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 13:06:31,160 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase9.apache.org,45203,1689339989183, sessionid=0x10163e34d0a0000, setting cluster-up flag (Was=false) 2023-07-14 13:06:31,177 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 13:06:31,182 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-14 13:06:31,183 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,45203,1689339989183 2023-07-14 13:06:31,192 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 13:06:31,203 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-14 13:06:31,204 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,45203,1689339989183 2023-07-14 13:06:31,206 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/.hbase-snapshot/.tmp 2023-07-14 13:06:31,294 INFO [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(951): ClusterId : 88dab552-822c-46d4-9648-1529c38da56c 2023-07-14 13:06:31,294 INFO [RS:1;jenkins-hbase9:38891] regionserver.HRegionServer(951): ClusterId : 88dab552-822c-46d4-9648-1529c38da56c 2023-07-14 13:06:31,299 DEBUG [RS:1;jenkins-hbase9:38891] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-14 13:06:31,299 DEBUG [RS:0;jenkins-hbase9:33743] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-14 13:06:31,305 DEBUG [RS:0;jenkins-hbase9:33743] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-14 13:06:31,305 DEBUG [RS:1;jenkins-hbase9:38891] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-14 13:06:31,305 DEBUG [RS:0;jenkins-hbase9:33743] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-14 13:06:31,305 DEBUG [RS:1;jenkins-hbase9:38891] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-14 13:06:31,309 DEBUG [RS:0;jenkins-hbase9:33743] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-14 13:06:31,309 DEBUG [RS:1;jenkins-hbase9:38891] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-14 13:06:31,310 DEBUG [RS:0;jenkins-hbase9:33743] zookeeper.ReadOnlyZKClient(139): Connect 0x7c2bb844 to 127.0.0.1:60854 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 13:06:31,310 DEBUG [RS:1;jenkins-hbase9:38891] zookeeper.ReadOnlyZKClient(139): Connect 0x4632495b to 127.0.0.1:60854 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 13:06:31,315 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-14 13:06:31,320 DEBUG [RS:0;jenkins-hbase9:33743] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72ec59b1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 13:06:31,321 DEBUG [RS:0;jenkins-hbase9:33743] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76d3d334, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-14 13:06:31,322 DEBUG [RS:1;jenkins-hbase9:38891] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48f5bfed, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 13:06:31,322 DEBUG [RS:1;jenkins-hbase9:38891] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cca222b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-14 13:06:31,327 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-14 13:06:31,327 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-14 13:06:31,327 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-14 13:06:31,327 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-14 13:06:31,328 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-07-14 13:06:31,328 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,328 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-14 13:06:31,328 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,330 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689340021330 2023-07-14 13:06:31,331 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-14 13:06:31,336 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-14 13:06:31,336 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-14 13:06:31,341 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-14 13:06:31,342 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-14 13:06:31,344 DEBUG [RS:1;jenkins-hbase9:38891] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase9:38891 2023-07-14 13:06:31,345 DEBUG [RS:0;jenkins-hbase9:33743] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase9:33743 2023-07-14 13:06:31,348 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-14 13:06:31,348 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-14 13:06:31,349 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-14 13:06:31,349 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-14 13:06:31,349 INFO [RS:0;jenkins-hbase9:33743] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-14 13:06:31,349 INFO [RS:1;jenkins-hbase9:38891] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-14 13:06:31,350 INFO [RS:1;jenkins-hbase9:38891] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-14 13:06:31,349 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:31,350 DEBUG [RS:1;jenkins-hbase9:38891] regionserver.HRegionServer(1022): About to register with Master. 2023-07-14 13:06:31,350 INFO [RS:0;jenkins-hbase9:33743] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-14 13:06:31,350 DEBUG [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(1022): About to register with Master. 2023-07-14 13:06:31,351 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-14 13:06:31,352 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-14 13:06:31,353 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-14 13:06:31,353 INFO [RS:1;jenkins-hbase9:38891] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase9.apache.org,45203,1689339989183 with isa=jenkins-hbase9.apache.org/172.31.2.10:38891, startcode=1689339990374 2023-07-14 13:06:31,353 INFO [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase9.apache.org,45203,1689339989183 with isa=jenkins-hbase9.apache.org/172.31.2.10:33743, startcode=1689339990331 2023-07-14 13:06:31,355 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-14 13:06:31,355 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-14 13:06:31,357 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689339991356,5,FailOnTimeoutGroup] 2023-07-14 13:06:31,357 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689339991357,5,FailOnTimeoutGroup] 2023-07-14 13:06:31,357 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:31,357 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-14 13:06:31,359 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:31,359 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:31,375 DEBUG [RS:0;jenkins-hbase9:33743] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-14 13:06:31,375 DEBUG [RS:1;jenkins-hbase9:38891] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-14 13:06:31,383 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-14 13:06:31,384 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-14 13:06:31,384 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf 2023-07-14 13:06:31,402 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 13:06:31,405 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-14 13:06:31,408 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/info 2023-07-14 13:06:31,409 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-14 13:06:31,412 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 13:06:31,413 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-14 13:06:31,416 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/rep_barrier 2023-07-14 13:06:31,416 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-14 13:06:31,417 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 13:06:31,417 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-14 13:06:31,421 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/table 2023-07-14 13:06:31,421 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-14 13:06:31,422 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 13:06:31,424 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740 2023-07-14 13:06:31,426 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740 2023-07-14 13:06:31,431 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-14 13:06:31,434 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-14 13:06:31,438 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-14 13:06:31,439 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61311488, jitterRate=-0.08638763427734375}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-14 13:06:31,439 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:48567, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-14 13:06:31,439 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:41191, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-14 13:06:31,439 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-14 13:06:31,440 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-14 13:06:31,440 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-14 13:06:31,440 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-14 13:06:31,440 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-14 13:06:31,440 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-14 13:06:31,441 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-14 13:06:31,441 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-14 13:06:31,447 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-14 13:06:31,447 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-14 13:06:31,450 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45203] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,38891,1689339990374 2023-07-14 13:06:31,454 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-14 13:06:31,460 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45203] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:31,465 DEBUG [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf 2023-07-14 13:06:31,465 DEBUG [RS:1;jenkins-hbase9:38891] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf 2023-07-14 13:06:31,465 DEBUG [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:34955 2023-07-14 13:06:31,466 DEBUG [RS:1;jenkins-hbase9:38891] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:34955 2023-07-14 13:06:31,466 DEBUG [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-14 13:06:31,466 DEBUG [RS:1;jenkins-hbase9:38891] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-14 13:06:31,467 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-14 13:06:31,469 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-14 13:06:31,471 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 13:06:31,475 DEBUG [RS:1;jenkins-hbase9:38891] zookeeper.ZKUtil(162): regionserver:38891-0x10163e34d0a0002, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,38891,1689339990374 2023-07-14 13:06:31,475 WARN [RS:1;jenkins-hbase9:38891] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-14 13:06:31,476 INFO [RS:1;jenkins-hbase9:38891] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 13:06:31,476 DEBUG [RS:0;jenkins-hbase9:33743] zookeeper.ZKUtil(162): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:31,476 DEBUG [RS:1;jenkins-hbase9:38891] regionserver.HRegionServer(1948): logDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374 2023-07-14 13:06:31,476 WARN [RS:0;jenkins-hbase9:33743] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-14 13:06:31,476 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,33743,1689339990331] 2023-07-14 13:06:31,477 INFO [RS:0;jenkins-hbase9:33743] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 13:06:31,477 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,38891,1689339990374] 2023-07-14 13:06:31,477 DEBUG [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(1948): logDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:31,485 DEBUG [RS:1;jenkins-hbase9:38891] zookeeper.ZKUtil(162): regionserver:38891-0x10163e34d0a0002, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:31,485 DEBUG [RS:0;jenkins-hbase9:33743] zookeeper.ZKUtil(162): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:31,486 DEBUG [RS:1;jenkins-hbase9:38891] zookeeper.ZKUtil(162): regionserver:38891-0x10163e34d0a0002, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,38891,1689339990374 2023-07-14 13:06:31,486 DEBUG [RS:0;jenkins-hbase9:33743] zookeeper.ZKUtil(162): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,38891,1689339990374 2023-07-14 13:06:31,496 DEBUG [RS:0;jenkins-hbase9:33743] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-14 13:06:31,496 DEBUG [RS:1;jenkins-hbase9:38891] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-14 13:06:31,505 INFO [RS:1;jenkins-hbase9:38891] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-14 13:06:31,505 INFO [RS:0;jenkins-hbase9:33743] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-14 13:06:31,524 INFO [RS:1;jenkins-hbase9:38891] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-14 13:06:31,524 INFO [RS:0;jenkins-hbase9:33743] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-14 13:06:31,528 INFO [RS:0;jenkins-hbase9:33743] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-14 13:06:31,528 INFO [RS:1;jenkins-hbase9:38891] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-14 13:06:31,528 INFO [RS:0;jenkins-hbase9:33743] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:31,528 INFO [RS:1;jenkins-hbase9:38891] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:31,530 INFO [RS:1;jenkins-hbase9:38891] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-14 13:06:31,530 INFO [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-14 13:06:31,537 INFO [RS:1;jenkins-hbase9:38891] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:31,537 INFO [RS:0;jenkins-hbase9:33743] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:31,537 DEBUG [RS:1;jenkins-hbase9:38891] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,537 DEBUG [RS:0;jenkins-hbase9:33743] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,537 DEBUG [RS:1;jenkins-hbase9:38891] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,537 DEBUG [RS:0;jenkins-hbase9:33743] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,537 DEBUG [RS:1;jenkins-hbase9:38891] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,537 DEBUG [RS:0;jenkins-hbase9:33743] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,538 DEBUG [RS:1;jenkins-hbase9:38891] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,538 DEBUG [RS:0;jenkins-hbase9:33743] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,538 DEBUG [RS:1;jenkins-hbase9:38891] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,538 DEBUG [RS:0;jenkins-hbase9:33743] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,538 DEBUG [RS:1;jenkins-hbase9:38891] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-14 13:06:31,538 DEBUG [RS:0;jenkins-hbase9:33743] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-14 13:06:31,538 DEBUG [RS:1;jenkins-hbase9:38891] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,538 DEBUG [RS:0;jenkins-hbase9:33743] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,538 DEBUG [RS:1;jenkins-hbase9:38891] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,538 DEBUG [RS:0;jenkins-hbase9:33743] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,538 DEBUG [RS:1;jenkins-hbase9:38891] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,538 DEBUG [RS:0;jenkins-hbase9:33743] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,538 DEBUG [RS:1;jenkins-hbase9:38891] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,538 DEBUG [RS:0;jenkins-hbase9:33743] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:31,539 INFO [RS:0;jenkins-hbase9:33743] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:31,539 INFO [RS:1;jenkins-hbase9:38891] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:31,540 INFO [RS:0;jenkins-hbase9:33743] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:31,540 INFO [RS:1;jenkins-hbase9:38891] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:31,540 INFO [RS:0;jenkins-hbase9:33743] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:31,540 INFO [RS:1;jenkins-hbase9:38891] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:31,555 INFO [RS:0;jenkins-hbase9:33743] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-14 13:06:31,555 INFO [RS:1;jenkins-hbase9:38891] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-14 13:06:31,557 INFO [RS:0;jenkins-hbase9:33743] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,33743,1689339990331-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:31,557 INFO [RS:1;jenkins-hbase9:38891] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,38891,1689339990374-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:31,574 INFO [RS:0;jenkins-hbase9:33743] regionserver.Replication(203): jenkins-hbase9.apache.org,33743,1689339990331 started 2023-07-14 13:06:31,574 INFO [RS:1;jenkins-hbase9:38891] regionserver.Replication(203): jenkins-hbase9.apache.org,38891,1689339990374 started 2023-07-14 13:06:31,574 INFO [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(1637): Serving as jenkins-hbase9.apache.org,33743,1689339990331, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:33743, sessionid=0x10163e34d0a0001 2023-07-14 13:06:31,574 INFO [RS:1;jenkins-hbase9:38891] regionserver.HRegionServer(1637): Serving as jenkins-hbase9.apache.org,38891,1689339990374, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:38891, sessionid=0x10163e34d0a0002 2023-07-14 13:06:31,575 DEBUG [RS:0;jenkins-hbase9:33743] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-14 13:06:31,575 DEBUG [RS:1;jenkins-hbase9:38891] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-14 13:06:31,575 DEBUG [RS:0;jenkins-hbase9:33743] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:31,575 DEBUG [RS:1;jenkins-hbase9:38891] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,38891,1689339990374 2023-07-14 13:06:31,576 DEBUG [RS:0;jenkins-hbase9:33743] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33743,1689339990331' 2023-07-14 13:06:31,576 DEBUG [RS:1;jenkins-hbase9:38891] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,38891,1689339990374' 2023-07-14 13:06:31,576 DEBUG [RS:1;jenkins-hbase9:38891] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-14 13:06:31,576 DEBUG [RS:0;jenkins-hbase9:33743] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-14 13:06:31,577 DEBUG [RS:1;jenkins-hbase9:38891] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-14 13:06:31,577 DEBUG [RS:0;jenkins-hbase9:33743] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-14 13:06:31,578 DEBUG [RS:0;jenkins-hbase9:33743] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-14 13:06:31,578 DEBUG [RS:1;jenkins-hbase9:38891] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-14 13:06:31,578 DEBUG [RS:0;jenkins-hbase9:33743] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-14 13:06:31,578 DEBUG [RS:1;jenkins-hbase9:38891] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-14 13:06:31,578 DEBUG [RS:0;jenkins-hbase9:33743] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:31,578 DEBUG [RS:1;jenkins-hbase9:38891] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,38891,1689339990374 2023-07-14 13:06:31,579 DEBUG [RS:1;jenkins-hbase9:38891] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,38891,1689339990374' 2023-07-14 13:06:31,579 DEBUG [RS:1;jenkins-hbase9:38891] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-14 13:06:31,579 DEBUG [RS:0;jenkins-hbase9:33743] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33743,1689339990331' 2023-07-14 13:06:31,579 DEBUG [RS:0;jenkins-hbase9:33743] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-14 13:06:31,579 DEBUG [RS:1;jenkins-hbase9:38891] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-14 13:06:31,580 DEBUG [RS:0;jenkins-hbase9:33743] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-14 13:06:31,580 DEBUG [RS:1;jenkins-hbase9:38891] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-14 13:06:31,580 DEBUG [RS:0;jenkins-hbase9:33743] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-14 13:06:31,580 INFO [RS:1;jenkins-hbase9:38891] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-14 13:06:31,580 INFO [RS:0;jenkins-hbase9:33743] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-14 13:06:31,581 INFO [RS:1;jenkins-hbase9:38891] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-14 13:06:31,581 INFO [RS:0;jenkins-hbase9:33743] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-14 13:06:31,621 DEBUG [jenkins-hbase9:45203] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-14 13:06:31,625 DEBUG [jenkins-hbase9:45203] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-07-14 13:06:31,630 DEBUG [jenkins-hbase9:45203] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 13:06:31,630 DEBUG [jenkins-hbase9:45203] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-14 13:06:31,630 DEBUG [jenkins-hbase9:45203] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-14 13:06:31,633 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,33743,1689339990331, state=OPENING 2023-07-14 13:06:31,640 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-14 13:06:31,641 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 13:06:31,642 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 13:06:31,645 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,33743,1689339990331}] 2023-07-14 13:06:31,691 INFO [RS:0;jenkins-hbase9:33743] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C33743%2C1689339990331, suffix=, logDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331, archiveDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/oldWALs, maxLogs=32 2023-07-14 13:06:31,691 INFO [RS:1;jenkins-hbase9:38891] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C38891%2C1689339990374, suffix=, logDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374, archiveDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/oldWALs, maxLogs=32 2023-07-14 13:06:31,712 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44567,DS-d1fe6f1d-a2c6-4e10-b2ab-1a91df578e1c,DISK] 2023-07-14 13:06:31,713 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45715,DS-d0e9d4fb-a902-4f72-9d50-25e21bea1880,DISK] 2023-07-14 13:06:31,713 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44567,DS-d1fe6f1d-a2c6-4e10-b2ab-1a91df578e1c,DISK] 2023-07-14 13:06:31,713 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45715,DS-d0e9d4fb-a902-4f72-9d50-25e21bea1880,DISK] 2023-07-14 13:06:31,724 INFO [RS:0;jenkins-hbase9:33743] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331/jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695 2023-07-14 13:06:31,724 INFO [RS:1;jenkins-hbase9:38891] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374/jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694 2023-07-14 13:06:31,724 DEBUG [RS:0;jenkins-hbase9:33743] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44567,DS-d1fe6f1d-a2c6-4e10-b2ab-1a91df578e1c,DISK], DatanodeInfoWithStorage[127.0.0.1:45715,DS-d0e9d4fb-a902-4f72-9d50-25e21bea1880,DISK]] 2023-07-14 13:06:31,725 DEBUG [RS:1;jenkins-hbase9:38891] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45715,DS-d0e9d4fb-a902-4f72-9d50-25e21bea1880,DISK], DatanodeInfoWithStorage[127.0.0.1:44567,DS-d1fe6f1d-a2c6-4e10-b2ab-1a91df578e1c,DISK]] 2023-07-14 13:06:31,826 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:31,829 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-14 13:06:31,832 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:44122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-14 13:06:31,842 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-14 13:06:31,842 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 13:06:31,847 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta, suffix=.meta, logDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331, archiveDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/oldWALs, maxLogs=32 2023-07-14 13:06:31,864 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44567,DS-d1fe6f1d-a2c6-4e10-b2ab-1a91df578e1c,DISK] 2023-07-14 13:06:31,865 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45715,DS-d0e9d4fb-a902-4f72-9d50-25e21bea1880,DISK] 2023-07-14 13:06:31,872 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331/jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta 2023-07-14 13:06:31,872 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44567,DS-d1fe6f1d-a2c6-4e10-b2ab-1a91df578e1c,DISK], DatanodeInfoWithStorage[127.0.0.1:45715,DS-d0e9d4fb-a902-4f72-9d50-25e21bea1880,DISK]] 2023-07-14 13:06:31,873 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-14 13:06:31,875 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-14 13:06:31,896 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-14 13:06:31,921 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-14 13:06:31,926 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-14 13:06:31,927 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 13:06:31,927 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-14 13:06:31,927 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-14 13:06:31,930 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-14 13:06:31,932 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/info 2023-07-14 13:06:31,932 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/info 2023-07-14 13:06:31,933 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-14 13:06:31,934 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 13:06:31,934 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-14 13:06:31,935 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/rep_barrier 2023-07-14 13:06:31,935 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/rep_barrier 2023-07-14 13:06:31,936 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-14 13:06:31,936 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 13:06:31,937 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-14 13:06:31,938 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/table 2023-07-14 13:06:31,938 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/table 2023-07-14 13:06:31,938 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-14 13:06:31,939 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 13:06:31,941 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740 2023-07-14 13:06:31,943 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740 2023-07-14 13:06:31,947 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-14 13:06:31,950 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-14 13:06:31,952 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63490218, jitterRate=-0.053922027349472046}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-14 13:06:31,952 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-14 13:06:31,961 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689339991819 2023-07-14 13:06:31,977 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-14 13:06:31,977 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-14 13:06:31,978 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,33743,1689339990331, state=OPEN 2023-07-14 13:06:31,981 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-14 13:06:31,981 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 13:06:31,987 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-14 13:06:31,987 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,33743,1689339990331 in 336 msec 2023-07-14 13:06:31,991 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-14 13:06:31,991 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 533 msec 2023-07-14 13:06:31,996 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 747 msec 2023-07-14 13:06:31,996 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689339991996, completionTime=-1 2023-07-14 13:06:31,996 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-14 13:06:31,997 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-14 13:06:32,058 DEBUG [hconnection-0x671abfa6-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 13:06:32,061 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:44128, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 13:06:32,076 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-14 13:06:32,076 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689340052076 2023-07-14 13:06:32,076 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689340112076 2023-07-14 13:06:32,076 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 79 msec 2023-07-14 13:06:32,097 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,45203,1689339989183-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:32,097 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,45203,1689339989183-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:32,098 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,45203,1689339989183-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:32,099 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:45203, period=300000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:32,100 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:32,105 DEBUG [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-14 13:06:32,113 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-14 13:06:32,114 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-14 13:06:32,123 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-14 13:06:32,125 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-14 13:06:32,127 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-14 13:06:32,148 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/.tmp/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:32,150 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/.tmp/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c empty. 2023-07-14 13:06:32,151 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/.tmp/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:32,151 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-14 13:06:32,176 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-14 13:06:32,178 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 96683f9c76c93ef3cd5418f55c4f230c, NAME => 'hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/.tmp 2023-07-14 13:06:32,193 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 13:06:32,193 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 96683f9c76c93ef3cd5418f55c4f230c, disabling compactions & flushes 2023-07-14 13:06:32,194 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:32,194 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:32,194 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. after waiting 0 ms 2023-07-14 13:06:32,194 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:32,194 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:32,194 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 96683f9c76c93ef3cd5418f55c4f230c: 2023-07-14 13:06:32,197 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-14 13:06:32,211 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689339992200"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689339992200"}]},"ts":"1689339992200"} 2023-07-14 13:06:32,235 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-14 13:06:32,237 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-14 13:06:32,241 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689339992237"}]},"ts":"1689339992237"} 2023-07-14 13:06:32,246 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-14 13:06:32,250 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-07-14 13:06:32,251 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 13:06:32,251 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-14 13:06:32,251 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-14 13:06:32,253 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=96683f9c76c93ef3cd5418f55c4f230c, ASSIGN}] 2023-07-14 13:06:32,256 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=96683f9c76c93ef3cd5418f55c4f230c, ASSIGN 2023-07-14 13:06:32,258 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=96683f9c76c93ef3cd5418f55c4f230c, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,33743,1689339990331; forceNewPlan=false, retain=false 2023-07-14 13:06:32,410 INFO [jenkins-hbase9:45203] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-14 13:06:32,411 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=96683f9c76c93ef3cd5418f55c4f230c, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:32,411 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689339992411"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689339992411"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689339992411"}]},"ts":"1689339992411"} 2023-07-14 13:06:32,416 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 96683f9c76c93ef3cd5418f55c4f230c, server=jenkins-hbase9.apache.org,33743,1689339990331}] 2023-07-14 13:06:32,575 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:32,576 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 96683f9c76c93ef3cd5418f55c4f230c, NAME => 'hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c.', STARTKEY => '', ENDKEY => ''} 2023-07-14 13:06:32,576 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:32,577 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 13:06:32,577 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:32,577 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:32,580 INFO [StoreOpener-96683f9c76c93ef3cd5418f55c4f230c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:32,582 DEBUG [StoreOpener-96683f9c76c93ef3cd5418f55c4f230c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/info 2023-07-14 13:06:32,582 DEBUG [StoreOpener-96683f9c76c93ef3cd5418f55c4f230c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/info 2023-07-14 13:06:32,582 INFO [StoreOpener-96683f9c76c93ef3cd5418f55c4f230c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96683f9c76c93ef3cd5418f55c4f230c columnFamilyName info 2023-07-14 13:06:32,583 INFO [StoreOpener-96683f9c76c93ef3cd5418f55c4f230c-1] regionserver.HStore(310): Store=96683f9c76c93ef3cd5418f55c4f230c/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 13:06:32,585 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:32,586 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:32,590 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:32,594 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-14 13:06:32,594 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 96683f9c76c93ef3cd5418f55c4f230c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70376722, jitterRate=0.048694878816604614}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 13:06:32,595 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 96683f9c76c93ef3cd5418f55c4f230c: 2023-07-14 13:06:32,597 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c., pid=6, masterSystemTime=1689339992569 2023-07-14 13:06:32,600 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:32,600 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:32,602 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=96683f9c76c93ef3cd5418f55c4f230c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:32,602 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689339992601"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689339992601"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689339992601"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689339992601"}]},"ts":"1689339992601"} 2023-07-14 13:06:32,610 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-14 13:06:32,610 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 96683f9c76c93ef3cd5418f55c4f230c, server=jenkins-hbase9.apache.org,33743,1689339990331 in 191 msec 2023-07-14 13:06:32,614 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-14 13:06:32,614 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=96683f9c76c93ef3cd5418f55c4f230c, ASSIGN in 357 msec 2023-07-14 13:06:32,615 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-14 13:06:32,615 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689339992615"}]},"ts":"1689339992615"} 2023-07-14 13:06:32,618 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-14 13:06:32,621 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-14 13:06:32,624 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 506 msec 2023-07-14 13:06:32,625 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-14 13:06:32,626 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-14 13:06:32,626 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 13:06:32,657 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-14 13:06:32,671 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-14 13:06:32,677 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 26 msec 2023-07-14 13:06:32,683 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-14 13:06:32,695 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-14 13:06:32,699 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-07-14 13:06:32,708 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-14 13:06:32,712 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-14 13:06:32,712 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.286sec 2023-07-14 13:06:32,714 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-14 13:06:32,716 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-14 13:06:32,716 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-14 13:06:32,717 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,45203,1689339989183-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-14 13:06:32,718 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,45203,1689339989183-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-14 13:06:32,726 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-14 13:06:32,799 DEBUG [Listener at localhost/40033] zookeeper.ReadOnlyZKClient(139): Connect 0x38241035 to 127.0.0.1:60854 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 13:06:32,804 DEBUG [Listener at localhost/40033] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@cceefda, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 13:06:32,817 DEBUG [hconnection-0x455e9d85-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 13:06:32,828 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:44144, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 13:06:32,838 INFO [Listener at localhost/40033] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase9.apache.org,45203,1689339989183 2023-07-14 13:06:32,847 DEBUG [Listener at localhost/40033] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-14 13:06:32,850 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:55192, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-14 13:06:32,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$4(2112): Client=jenkins//172.31.2.10 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-14 13:06:32,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-14 13:06:32,865 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-14 13:06:32,867 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-14 13:06:32,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(700): Client=jenkins//172.31.2.10 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-14 13:06:32,869 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/.tmp/data/default/t/148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:32,870 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/.tmp/data/default/t/148d9abb69f2f1f614322f28262c0164 empty. 2023-07-14 13:06:32,872 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/.tmp/data/default/t/148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:32,872 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-14 13:06:32,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-14 13:06:32,891 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-14 13:06:32,893 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => 148d9abb69f2f1f614322f28262c0164, NAME => 't,,1689339992857.148d9abb69f2f1f614322f28262c0164.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/.tmp 2023-07-14 13:06:32,907 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689339992857.148d9abb69f2f1f614322f28262c0164.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 13:06:32,907 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing 148d9abb69f2f1f614322f28262c0164, disabling compactions & flushes 2023-07-14 13:06:32,907 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:32,907 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:32,908 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689339992857.148d9abb69f2f1f614322f28262c0164. after waiting 0 ms 2023-07-14 13:06:32,908 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:32,908 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:32,908 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for 148d9abb69f2f1f614322f28262c0164: 2023-07-14 13:06:32,912 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-14 13:06:32,913 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689339992857.148d9abb69f2f1f614322f28262c0164.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689339992913"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689339992913"}]},"ts":"1689339992913"} 2023-07-14 13:06:32,916 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-14 13:06:32,917 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-14 13:06:32,917 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689339992917"}]},"ts":"1689339992917"} 2023-07-14 13:06:32,919 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-14 13:06:32,928 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-07-14 13:06:32,929 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 13:06:32,929 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-14 13:06:32,929 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-14 13:06:32,930 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=148d9abb69f2f1f614322f28262c0164, ASSIGN}] 2023-07-14 13:06:32,931 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=148d9abb69f2f1f614322f28262c0164, ASSIGN 2023-07-14 13:06:32,933 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=148d9abb69f2f1f614322f28262c0164, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,38891,1689339990374; forceNewPlan=false, retain=false 2023-07-14 13:06:32,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-14 13:06:33,083 INFO [jenkins-hbase9:45203] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-14 13:06:33,084 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=148d9abb69f2f1f614322f28262c0164, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,38891,1689339990374 2023-07-14 13:06:33,084 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689339992857.148d9abb69f2f1f614322f28262c0164.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689339993084"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689339993084"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689339993084"}]},"ts":"1689339993084"} 2023-07-14 13:06:33,087 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 148d9abb69f2f1f614322f28262c0164, server=jenkins-hbase9.apache.org,38891,1689339990374}] 2023-07-14 13:06:33,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-14 13:06:33,241 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,38891,1689339990374 2023-07-14 13:06:33,241 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-14 13:06:33,244 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:40446, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-14 13:06:33,249 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:33,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 148d9abb69f2f1f614322f28262c0164, NAME => 't,,1689339992857.148d9abb69f2f1f614322f28262c0164.', STARTKEY => '', ENDKEY => ''} 2023-07-14 13:06:33,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:33,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated t,,1689339992857.148d9abb69f2f1f614322f28262c0164.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 13:06:33,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:33,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:33,252 INFO [StoreOpener-148d9abb69f2f1f614322f28262c0164-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:33,254 DEBUG [StoreOpener-148d9abb69f2f1f614322f28262c0164-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/f 2023-07-14 13:06:33,254 DEBUG [StoreOpener-148d9abb69f2f1f614322f28262c0164-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/f 2023-07-14 13:06:33,255 INFO [StoreOpener-148d9abb69f2f1f614322f28262c0164-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 148d9abb69f2f1f614322f28262c0164 columnFamilyName f 2023-07-14 13:06:33,256 INFO [StoreOpener-148d9abb69f2f1f614322f28262c0164-1] regionserver.HStore(310): Store=148d9abb69f2f1f614322f28262c0164/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 13:06:33,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:33,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:33,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:33,266 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-14 13:06:33,267 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 148d9abb69f2f1f614322f28262c0164; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70359869, jitterRate=0.0484437495470047}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 13:06:33,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 148d9abb69f2f1f614322f28262c0164: 2023-07-14 13:06:33,268 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689339992857.148d9abb69f2f1f614322f28262c0164., pid=11, masterSystemTime=1689339993241 2023-07-14 13:06:33,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:33,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:33,272 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=148d9abb69f2f1f614322f28262c0164, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,38891,1689339990374 2023-07-14 13:06:33,273 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689339992857.148d9abb69f2f1f614322f28262c0164.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689339993272"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689339993272"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689339993272"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689339993272"}]},"ts":"1689339993272"} 2023-07-14 13:06:33,278 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-14 13:06:33,278 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 148d9abb69f2f1f614322f28262c0164, server=jenkins-hbase9.apache.org,38891,1689339990374 in 188 msec 2023-07-14 13:06:33,281 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-14 13:06:33,281 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=148d9abb69f2f1f614322f28262c0164, ASSIGN in 348 msec 2023-07-14 13:06:33,282 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-14 13:06:33,282 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689339993282"}]},"ts":"1689339993282"} 2023-07-14 13:06:33,284 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-14 13:06:33,287 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-14 13:06:33,290 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 426 msec 2023-07-14 13:06:33,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-14 13:06:33,486 INFO [Listener at localhost/40033] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-14 13:06:33,486 DEBUG [Listener at localhost/40033] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-14 13:06:33,488 INFO [Listener at localhost/40033] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-14 13:06:33,492 INFO [Listener at localhost/40033] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-14 13:06:33,492 INFO [Listener at localhost/40033] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-14 13:06:33,492 INFO [Listener at localhost/40033] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-14 13:06:33,495 DEBUG [Listener at localhost/40033] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 13:06:33,499 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:40454, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 13:06:33,543 INFO [Listener at localhost/40033] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=294, OpenFileDescriptor=585, MaxFileDescriptor=60000, SystemLoadAverage=136, ProcessCount=174, AvailableMemoryMB=11441 2023-07-14 13:06:33,544 INFO [Listener at localhost/40033] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-14 13:06:33,547 DEBUG [Listener at localhost/40033] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-14 13:06:33,548 DEBUG [Listener at localhost/40033] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:33,548 INFO [Listener at localhost/40033] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-14 13:06:33,548 INFO [Listener at localhost/40033] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-14 13:06:33,551 INFO [Listener at localhost/40033] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689339992615/Put/vlen=2/seqid=0} 2023-07-14 13:06:33,552 INFO [Listener at localhost/40033] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-14 13:06:33,552 INFO [Listener at localhost/40033] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c./info:regioninfo/1689339992601/Put/vlen=41/seqid=0, hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c./info:seqnumDuringOpen/1689339992601/Put/vlen=8/seqid=0, hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c./info:server/1689339992601/Put/vlen=31/seqid=0, hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c./info:serverstartcode/1689339992601/Put/vlen=8/seqid=0, hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c./info:sn/1689339992411/Put/vlen=45/seqid=0, hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c./info:state/1689339992601/Put/vlen=4/seqid=0} 2023-07-14 13:06:33,552 INFO [Listener at localhost/40033] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 96683f9c76c93ef3cd5418f55c4f230c, NAME => 'hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c.', STARTKEY => '', ENDKEY => ''} 2023-07-14 13:06:33,552 INFO [Listener at localhost/40033] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689339993282/Put/vlen=2/seqid=0} 2023-07-14 13:06:33,552 INFO [Listener at localhost/40033] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-14 13:06:33,552 INFO [Listener at localhost/40033] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689339992857.148d9abb69f2f1f614322f28262c0164./info:regioninfo/1689339993272/Put/vlen=35/seqid=0, t,,1689339992857.148d9abb69f2f1f614322f28262c0164./info:seqnumDuringOpen/1689339993272/Put/vlen=8/seqid=0, t,,1689339992857.148d9abb69f2f1f614322f28262c0164./info:server/1689339993272/Put/vlen=31/seqid=0, t,,1689339992857.148d9abb69f2f1f614322f28262c0164./info:serverstartcode/1689339993272/Put/vlen=8/seqid=0, t,,1689339992857.148d9abb69f2f1f614322f28262c0164./info:sn/1689339993084/Put/vlen=45/seqid=0, t,,1689339992857.148d9abb69f2f1f614322f28262c0164./info:state/1689339993272/Put/vlen=4/seqid=0} 2023-07-14 13:06:33,553 INFO [Listener at localhost/40033] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 148d9abb69f2f1f614322f28262c0164, NAME => 't,,1689339992857.148d9abb69f2f1f614322f28262c0164.', STARTKEY => '', ENDKEY => ''} 2023-07-14 13:06:33,553 DEBUG [Listener at localhost/40033] zookeeper.ReadOnlyZKClient(139): Connect 0x4953d7ba to 127.0.0.1:60854 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 13:06:33,557 DEBUG [Listener at localhost/40033] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b7348c9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 13:06:33,558 INFO [Listener at localhost/40033] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-14 13:06:33,558 INFO [Listener at localhost/40033] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-14 13:06:33,560 DEBUG [hconnection-0x202b2c84-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 13:06:33,563 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:44158, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 13:06:33,567 DEBUG [hconnection-0x202b2c84-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 13:06:33,570 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:40458, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 13:06:33,571 INFO [Listener at localhost/40033] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-14 13:06:33,574 ERROR [Listener at localhost/40033] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase9.apache.org,38891,1689339990374: die! ***** 2023-07-14 13:06:33,575 ERROR [Listener at localhost/40033] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-14 13:06:33,577 DEBUG [Listener at localhost/40033] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-14 13:06:33,580 DEBUG [Listener at localhost/40033] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-14 13:06:33,580 DEBUG [Listener at localhost/40033] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-14 13:06:33,581 DEBUG [Listener at localhost/40033] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-14 13:06:33,582 INFO [Listener at localhost/40033] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 992477184, "init": 513802240, "max": 2051014656, "used": 424810832 }, "NonHeapMemoryUsage": { "committed": 112877568, "init": 2555904, "max": -1, "used": 110831344 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase9.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase9.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase9.apache.org" } ] } 2023-07-14 13:06:33,587 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45203] master.MasterRpcServices(612): jenkins-hbase9.apache.org,38891,1689339990374 reported a fatal error: ***** ABORTING region server jenkins-hbase9.apache.org,38891,1689339990374: die! ***** 2023-07-14 13:06:33,589 INFO [Listener at localhost/40033] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,38891,1689339990374' ***** 2023-07-14 13:06:33,590 INFO [Listener at localhost/40033] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-14 13:06:33,590 INFO [RS:1;jenkins-hbase9:38891] regionserver.HeapMemoryManager(220): Stopping 2023-07-14 13:06:33,591 INFO [RS:1;jenkins-hbase9:38891] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-14 13:06:33,591 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-14 13:06:33,591 INFO [RS:1;jenkins-hbase9:38891] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-14 13:06:33,591 INFO [RS:1;jenkins-hbase9:38891] regionserver.HRegionServer(3305): Received CLOSE for 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:33,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38891] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner 2505479681906573313 2023-07-14 13:06:33,591 INFO [RS:1;jenkins-hbase9:38891] regionserver.HRegionServer(1141): aborting server jenkins-hbase9.apache.org,38891,1689339990374 2023-07-14 13:06:33,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38891] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,38891,1689339990374 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 13:06:33,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 148d9abb69f2f1f614322f28262c0164, disabling compactions & flushes 2023-07-14 13:06:33,592 DEBUG [RS:1;jenkins-hbase9:38891] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4632495b to 127.0.0.1:60854 2023-07-14 13:06:33,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38891] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.2.10:40458 deadline: 1689340053591, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,38891,1689339990374 aborting 2023-07-14 13:06:33,592 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:33,592 DEBUG [RS:1;jenkins-hbase9:38891] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 13:06:33,593 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:33,593 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689339992857.148d9abb69f2f1f614322f28262c0164. after waiting 0 ms 2023-07-14 13:06:33,593 INFO [RS:1;jenkins-hbase9:38891] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-14 13:06:33,593 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:33,593 DEBUG [RS:1;jenkins-hbase9:38891] regionserver.HRegionServer(1478): Online Regions={148d9abb69f2f1f614322f28262c0164=t,,1689339992857.148d9abb69f2f1f614322f28262c0164.} 2023-07-14 13:06:33,594 DEBUG [RS:1;jenkins-hbase9:38891] regionserver.HRegionServer(1504): Waiting on 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:33,594 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:33,595 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:33,595 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 148d9abb69f2f1f614322f28262c0164: 2023-07-14 13:06:33,595 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:33,609 DEBUG [Listener at localhost/40033] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,38891,1689339990374 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,38891,1689339990374 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-14 13:06:33,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38891] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,38891,1689339990374 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 13:06:33,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38891] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.2.10:40458 deadline: 1689340053611, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,38891,1689339990374 aborting 2023-07-14 13:06:33,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38891] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,38891,1689339990374 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 13:06:33,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38891] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Scan size: 125 connection: 172.31.2.10:40458 deadline: 1689340053720, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,38891,1689339990374 aborting 2023-07-14 13:06:33,795 INFO [RS:1;jenkins-hbase9:38891] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,38891,1689339990374; all regions closed. 2023-07-14 13:06:33,804 DEBUG [RS:1;jenkins-hbase9:38891] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 13:06:33,805 INFO [RS:1;jenkins-hbase9:38891] regionserver.LeaseManager(133): Closed leases 2023-07-14 13:06:33,805 INFO [RS:1;jenkins-hbase9:38891] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-14 13:06:33,805 INFO [RS:1;jenkins-hbase9:38891] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-14 13:06:33,805 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-14 13:06:33,805 INFO [RS:1;jenkins-hbase9:38891] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-14 13:06:33,805 INFO [RS:1;jenkins-hbase9:38891] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-14 13:06:33,806 INFO [RS:1;jenkins-hbase9:38891] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:38891 2023-07-14 13:06:33,820 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:38891-0x10163e34d0a0002, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,38891,1689339990374 2023-07-14 13:06:33,820 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 13:06:33,820 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:38891-0x10163e34d0a0002, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 13:06:33,820 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,38891,1689339990374 2023-07-14 13:06:33,821 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 13:06:33,823 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,38891,1689339990374] 2023-07-14 13:06:33,823 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,38891,1689339990374; numProcessing=1 2023-07-14 13:06:33,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:33,823 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase9.apache.org,38891,1689339990374 znode expired, triggering replicatorRemoved event 2023-07-14 13:06:33,824 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,38891,1689339990374 already deleted, retry=false 2023-07-14 13:06:33,824 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase9.apache.org,38891,1689339990374 on jenkins-hbase9.apache.org,45203,1689339989183 2023-07-14 13:06:33,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:33,833 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase9.apache.org,38891,1689339990374, splitWal=true, meta=false 2023-07-14 13:06:33,834 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase9.apache.org,38891,1689339990374 (carryingMeta=false) jenkins-hbase9.apache.org,38891,1689339990374/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@724b714[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-14 13:06:33,836 INFO [PEWorker-4] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,38891,1689339990374, splitWal=true, meta=false 2023-07-14 13:06:33,838 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase9.apache.org,38891,1689339990374 had 1 regions 2023-07-14 13:06:33,840 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,38891,1689339990374, splitWal=true, meta=false, isMeta: false 2023-07-14 13:06:33,842 DEBUG [PEWorker-4] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374-splitting 2023-07-14 13:06:33,844 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase9.apache.org,38891,1689339990374 WAL count=1, meta=false 2023-07-14 13:06:33,847 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694}] 2023-07-14 13:06:33,855 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:33,856 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694, worker=jenkins-hbase9.apache.org,33743,1689339990331}] 2023-07-14 13:06:33,922 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:38891-0x10163e34d0a0002, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 13:06:33,923 INFO [RS:1;jenkins-hbase9:38891] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,38891,1689339990374; zookeeper connection closed. 2023-07-14 13:06:33,923 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:38891-0x10163e34d0a0002, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 13:06:33,923 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@304fdd1e] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@304fdd1e 2023-07-14 13:06:33,927 WARN [RPCClient-NioEventLoopGroup-5-5] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:38891 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:38891 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 13:06:33,928 DEBUG [RPCClient-NioEventLoopGroup-5-5] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:38891 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:38891 2023-07-14 13:06:34,020 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33743] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-14 13:06:34,041 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374-splitting/jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694, size=1.1 K (1148bytes) 2023-07-14 13:06:34,042 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374-splitting/jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694 2023-07-14 13:06:34,046 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374-splitting/jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694 after 4ms 2023-07-14 13:06:34,052 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(311): Open hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374-splitting/jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694 took 12ms 2023-07-14 13:06:34,066 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(346): Last flushed sequenceid for 148d9abb69f2f1f614322f28262c0164: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-14 13:06:34,068 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374-splitting/jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694 so closing down 2023-07-14 13:06:34,068 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-14 13:06:34,073 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-14 13:06:34,075 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694.temp 2023-07-14 13:06:34,076 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-14 13:06:34,087 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-14 13:06:34,090 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694.temp to hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/recovered.edits/0000000000000000013 2023-07-14 13:06:34,090 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 37 ms; skipped=1; WAL=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374-splitting/jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694, size=1.1 K, length=1148, corrupted=false, cancelled=false 2023-07-14 13:06:34,090 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374-splitting/jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694, journal: Splitting hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374-splitting/jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694, size=1.1 K (1148bytes) at 1689339994041Finishing writing output for hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374-splitting/jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694 so closing down at 1689339994068 (+27 ms)Creating recovered edits writer path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694.temp at 1689339994075 (+7 ms)3 split writer threads finished at 1689339994076 (+1 ms)Closed recovered edits writer path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689339994088 (+12 ms)Rename recovered edits hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694.temp to hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/recovered.edits/0000000000000000013 at 1689339994090 (+2 ms)Processed 11 edits across 1 Regions in 37 ms; skipped=1; WAL=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374-splitting/jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694, size=1.1 K, length=1148, corrupted=false, cancelled=false at 1689339994090 2023-07-14 13:06:34,091 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374-splitting/jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694 2023-07-14 13:06:34,093 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-14 13:06:34,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster(3641): Remote procedure done, pid=14 2023-07-14 13:06:34,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] wal.WALSplitUtil(141): Moved hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374-splitting/jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694 to hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/oldWALs 2023-07-14 13:06:34,105 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-14 13:06:34,105 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694, worker=jenkins-hbase9.apache.org,33743,1689339990331 in 247 msec 2023-07-14 13:06:34,107 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:34,110 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-14 13:06:34,110 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase9.apache.org%2C38891%2C1689339990374.1689339991694, worker=jenkins-hbase9.apache.org,33743,1689339990331 in 261 msec 2023-07-14 13:06:34,112 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374-splitting dir is empty, no logs to split. 2023-07-14 13:06:34,112 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase9.apache.org,38891,1689339990374 WAL count=0, meta=false 2023-07-14 13:06:34,112 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase9.apache.org,38891,1689339990374 WAL splitting is done? wals=0, meta=false 2023-07-14 13:06:34,115 INFO [PEWorker-4] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase9.apache.org,38891,1689339990374 failed, ignore...File hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,38891,1689339990374-splitting does not exist. 2023-07-14 13:06:34,117 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=148d9abb69f2f1f614322f28262c0164, ASSIGN}] 2023-07-14 13:06:34,118 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=148d9abb69f2f1f614322f28262c0164, ASSIGN 2023-07-14 13:06:34,119 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=148d9abb69f2f1f614322f28262c0164, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 13:06:34,235 DEBUG [hconnection-0x202b2c84-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase9.apache.org/172.31.2.10:38891 this server is in the failed servers list 2023-07-14 13:06:34,269 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=148d9abb69f2f1f614322f28262c0164, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:34,270 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689339992857.148d9abb69f2f1f614322f28262c0164.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689339994269"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689339994269"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689339994269"}]},"ts":"1689339994269"} 2023-07-14 13:06:34,272 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 148d9abb69f2f1f614322f28262c0164, server=jenkins-hbase9.apache.org,33743,1689339990331}] 2023-07-14 13:06:34,430 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:34,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 148d9abb69f2f1f614322f28262c0164, NAME => 't,,1689339992857.148d9abb69f2f1f614322f28262c0164.', STARTKEY => '', ENDKEY => ''} 2023-07-14 13:06:34,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:34,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated t,,1689339992857.148d9abb69f2f1f614322f28262c0164.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 13:06:34,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:34,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:34,432 INFO [StoreOpener-148d9abb69f2f1f614322f28262c0164-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:34,434 DEBUG [StoreOpener-148d9abb69f2f1f614322f28262c0164-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/f 2023-07-14 13:06:34,434 DEBUG [StoreOpener-148d9abb69f2f1f614322f28262c0164-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/f 2023-07-14 13:06:34,434 INFO [StoreOpener-148d9abb69f2f1f614322f28262c0164-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 148d9abb69f2f1f614322f28262c0164 columnFamilyName f 2023-07-14 13:06:34,435 INFO [StoreOpener-148d9abb69f2f1f614322f28262c0164-1] regionserver.HStore(310): Store=148d9abb69f2f1f614322f28262c0164/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 13:06:34,437 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:34,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:34,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/recovered.edits/0000000000000000013 2023-07-14 13:06:34,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/recovered.edits/0000000000000000013 2023-07-14 13:06:34,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 148d9abb69f2f1f614322f28262c0164 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-14 13:06:34,503 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/.tmp/f/f5dc6056b7ed4735861f59e27681b803 2023-07-14 13:06:34,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/.tmp/f/f5dc6056b7ed4735861f59e27681b803 as hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/f/f5dc6056b7ed4735861f59e27681b803 2023-07-14 13:06:34,542 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-14 13:06:34,549 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/f/f5dc6056b7ed4735861f59e27681b803, entries=10, sequenceid=13, filesize=4.8 K 2023-07-14 13:06:34,550 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for 148d9abb69f2f1f614322f28262c0164 in 102ms, sequenceid=13, compaction requested=false; wal=null 2023-07-14 13:06:34,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-14 13:06:34,554 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/recovered.edits/0000000000000000013 2023-07-14 13:06:34,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:34,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-14 13:06:34,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 148d9abb69f2f1f614322f28262c0164; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72745808, jitterRate=0.08399701118469238}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 13:06:34,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 148d9abb69f2f1f614322f28262c0164: 2023-07-14 13:06:34,563 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689339992857.148d9abb69f2f1f614322f28262c0164., pid=16, masterSystemTime=1689339994425 2023-07-14 13:06:34,565 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:34,565 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:34,566 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=148d9abb69f2f1f614322f28262c0164, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:34,567 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689339992857.148d9abb69f2f1f614322f28262c0164.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689339994566"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689339994566"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689339994566"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689339994566"}]},"ts":"1689339994566"} 2023-07-14 13:06:34,572 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-14 13:06:34,572 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 148d9abb69f2f1f614322f28262c0164, server=jenkins-hbase9.apache.org,33743,1689339990331 in 297 msec 2023-07-14 13:06:34,575 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-14 13:06:34,575 INFO [PEWorker-4] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase9.apache.org,38891,1689339990374 after splitting done 2023-07-14 13:06:34,575 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=148d9abb69f2f1f614322f28262c0164, ASSIGN in 455 msec 2023-07-14 13:06:34,575 DEBUG [PEWorker-4] master.DeadServer(114): Removed jenkins-hbase9.apache.org,38891,1689339990374 from processing; numProcessing=0 2023-07-14 13:06:34,577 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase9.apache.org,38891,1689339990374, splitWal=true, meta=false in 749 msec 2023-07-14 13:06:34,757 DEBUG [Listener at localhost/40033] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4953d7ba to 127.0.0.1:60854 2023-07-14 13:06:34,757 DEBUG [Listener at localhost/40033] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 13:06:34,757 INFO [Listener at localhost/40033] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-14 13:06:34,770 INFO [Listener at localhost/40033] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=270 (was 294), OpenFileDescriptor=567 (was 585), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=136 (was 136), ProcessCount=174 (was 174), AvailableMemoryMB=11434 (was 11441) 2023-07-14 13:06:34,780 INFO [Listener at localhost/40033] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=270, OpenFileDescriptor=567, MaxFileDescriptor=60000, SystemLoadAverage=136, ProcessCount=174, AvailableMemoryMB=11434 2023-07-14 13:06:34,781 INFO [Listener at localhost/40033] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase9:38891 2023-07-14 13:06:34,796 INFO [Listener at localhost/40033] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-07-14 13:06:34,796 INFO [Listener at localhost/40033] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 13:06:34,796 INFO [Listener at localhost/40033] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-14 13:06:34,796 INFO [Listener at localhost/40033] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-14 13:06:34,797 INFO [Listener at localhost/40033] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 13:06:34,797 INFO [Listener at localhost/40033] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-14 13:06:34,797 INFO [Listener at localhost/40033] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-14 13:06:34,799 INFO [Listener at localhost/40033] ipc.NettyRpcServer(120): Bind to /172.31.2.10:43229 2023-07-14 13:06:34,799 INFO [Listener at localhost/40033] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-14 13:06:34,800 DEBUG [Listener at localhost/40033] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-14 13:06:34,801 INFO [Listener at localhost/40033] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 13:06:34,803 INFO [Listener at localhost/40033] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 13:06:34,804 INFO [Listener at localhost/40033] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43229 connecting to ZooKeeper ensemble=127.0.0.1:60854 2023-07-14 13:06:34,809 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:432290x0, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-14 13:06:34,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:43229-0x10163e34d0a0008 connected 2023-07-14 13:06:34,811 DEBUG [Listener at localhost/40033] zookeeper.ZKUtil(162): regionserver:43229-0x10163e34d0a0008, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-14 13:06:34,811 DEBUG [Listener at localhost/40033] zookeeper.ZKUtil(162): regionserver:43229-0x10163e34d0a0008, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-14 13:06:34,812 DEBUG [Listener at localhost/40033] zookeeper.ZKUtil(164): regionserver:43229-0x10163e34d0a0008, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-14 13:06:34,813 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43229 2023-07-14 13:06:34,813 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43229 2023-07-14 13:06:34,813 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43229 2023-07-14 13:06:34,813 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43229 2023-07-14 13:06:34,813 DEBUG [Listener at localhost/40033] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43229 2023-07-14 13:06:34,815 INFO [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(951): ClusterId : 88dab552-822c-46d4-9648-1529c38da56c 2023-07-14 13:06:34,816 DEBUG [RS:2;jenkins-hbase9:43229] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-14 13:06:34,818 DEBUG [RS:2;jenkins-hbase9:43229] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-14 13:06:34,818 DEBUG [RS:2;jenkins-hbase9:43229] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-14 13:06:34,820 DEBUG [RS:2;jenkins-hbase9:43229] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-14 13:06:34,821 DEBUG [RS:2;jenkins-hbase9:43229] zookeeper.ReadOnlyZKClient(139): Connect 0x75643ef5 to 127.0.0.1:60854 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 13:06:34,825 DEBUG [RS:2;jenkins-hbase9:43229] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ba3de29, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 13:06:34,825 DEBUG [RS:2;jenkins-hbase9:43229] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f9c5585, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-14 13:06:34,836 DEBUG [RS:2;jenkins-hbase9:43229] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase9:43229 2023-07-14 13:06:34,836 INFO [RS:2;jenkins-hbase9:43229] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-14 13:06:34,836 INFO [RS:2;jenkins-hbase9:43229] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-14 13:06:34,836 DEBUG [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(1022): About to register with Master. 2023-07-14 13:06:34,837 INFO [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase9.apache.org,45203,1689339989183 with isa=jenkins-hbase9.apache.org/172.31.2.10:43229, startcode=1689339994795 2023-07-14 13:06:34,837 DEBUG [RS:2;jenkins-hbase9:43229] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-14 13:06:34,841 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:45127, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-14 13:06:34,842 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45203] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:34,842 DEBUG [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf 2023-07-14 13:06:34,842 DEBUG [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:34955 2023-07-14 13:06:34,842 DEBUG [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-14 13:06:34,845 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 13:06:34,845 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 13:06:34,845 DEBUG [RS:2;jenkins-hbase9:43229] zookeeper.ZKUtil(162): regionserver:43229-0x10163e34d0a0008, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:34,845 WARN [RS:2;jenkins-hbase9:43229] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-14 13:06:34,846 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,43229,1689339994795] 2023-07-14 13:06:34,846 INFO [RS:2;jenkins-hbase9:43229] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 13:06:34,846 DEBUG [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(1948): logDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:34,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:34,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:34,853 DEBUG [RS:2;jenkins-hbase9:43229] zookeeper.ZKUtil(162): regionserver:43229-0x10163e34d0a0008, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:34,854 DEBUG [RS:2;jenkins-hbase9:43229] zookeeper.ZKUtil(162): regionserver:43229-0x10163e34d0a0008, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:34,855 DEBUG [RS:2;jenkins-hbase9:43229] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-14 13:06:34,855 INFO [RS:2;jenkins-hbase9:43229] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-14 13:06:34,859 INFO [RS:2;jenkins-hbase9:43229] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-14 13:06:34,859 INFO [RS:2;jenkins-hbase9:43229] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-14 13:06:34,859 INFO [RS:2;jenkins-hbase9:43229] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:34,859 INFO [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-14 13:06:34,860 INFO [RS:2;jenkins-hbase9:43229] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:34,861 DEBUG [RS:2;jenkins-hbase9:43229] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:34,861 DEBUG [RS:2;jenkins-hbase9:43229] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:34,861 DEBUG [RS:2;jenkins-hbase9:43229] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:34,861 DEBUG [RS:2;jenkins-hbase9:43229] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:34,861 DEBUG [RS:2;jenkins-hbase9:43229] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:34,861 DEBUG [RS:2;jenkins-hbase9:43229] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-14 13:06:34,861 DEBUG [RS:2;jenkins-hbase9:43229] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:34,861 DEBUG [RS:2;jenkins-hbase9:43229] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:34,861 DEBUG [RS:2;jenkins-hbase9:43229] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:34,861 DEBUG [RS:2;jenkins-hbase9:43229] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-14 13:06:34,864 INFO [RS:2;jenkins-hbase9:43229] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:34,865 INFO [RS:2;jenkins-hbase9:43229] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:34,865 INFO [RS:2;jenkins-hbase9:43229] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:34,875 INFO [RS:2;jenkins-hbase9:43229] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-14 13:06:34,875 INFO [RS:2;jenkins-hbase9:43229] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,43229,1689339994795-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 13:06:34,886 INFO [RS:2;jenkins-hbase9:43229] regionserver.Replication(203): jenkins-hbase9.apache.org,43229,1689339994795 started 2023-07-14 13:06:34,886 INFO [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(1637): Serving as jenkins-hbase9.apache.org,43229,1689339994795, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:43229, sessionid=0x10163e34d0a0008 2023-07-14 13:06:34,886 INFO [Listener at localhost/40033] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase9:43229,5,FailOnTimeoutGroup] 2023-07-14 13:06:34,886 DEBUG [RS:2;jenkins-hbase9:43229] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-14 13:06:34,886 DEBUG [RS:2;jenkins-hbase9:43229] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:34,886 DEBUG [RS:2;jenkins-hbase9:43229] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,43229,1689339994795' 2023-07-14 13:06:34,886 DEBUG [RS:2;jenkins-hbase9:43229] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-14 13:06:34,886 INFO [Listener at localhost/40033] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase9.apache.org,38891,1689339990374 2023-07-14 13:06:34,886 INFO [Listener at localhost/40033] client.TestScannerTimeout(177): START ************ test3686b 2023-07-14 13:06:34,886 DEBUG [RS:2;jenkins-hbase9:43229] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-14 13:06:34,887 DEBUG [RS:2;jenkins-hbase9:43229] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-14 13:06:34,887 DEBUG [RS:2;jenkins-hbase9:43229] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-14 13:06:34,887 DEBUG [RS:2;jenkins-hbase9:43229] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:34,887 DEBUG [RS:2;jenkins-hbase9:43229] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,43229,1689339994795' 2023-07-14 13:06:34,887 DEBUG [RS:2;jenkins-hbase9:43229] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-14 13:06:34,887 DEBUG [RS:2;jenkins-hbase9:43229] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-14 13:06:34,888 DEBUG [RS:2;jenkins-hbase9:43229] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-14 13:06:34,888 INFO [RS:2;jenkins-hbase9:43229] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-14 13:06:34,888 INFO [RS:2;jenkins-hbase9:43229] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-14 13:06:34,889 DEBUG [Listener at localhost/40033] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-14 13:06:34,889 DEBUG [Listener at localhost/40033] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:34,890 DEBUG [Listener at localhost/40033] zookeeper.ReadOnlyZKClient(139): Connect 0x4d07ef26 to 127.0.0.1:60854 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 13:06:34,894 DEBUG [Listener at localhost/40033] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e01c86d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 13:06:34,896 DEBUG [hconnection-0x5ce8cbd2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 13:06:34,899 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:44168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 13:06:34,990 INFO [RS:2;jenkins-hbase9:43229] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C43229%2C1689339994795, suffix=, logDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,43229,1689339994795, archiveDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/oldWALs, maxLogs=32 2023-07-14 13:06:35,017 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44567,DS-d1fe6f1d-a2c6-4e10-b2ab-1a91df578e1c,DISK] 2023-07-14 13:06:35,017 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45715,DS-d0e9d4fb-a902-4f72-9d50-25e21bea1880,DISK] 2023-07-14 13:06:35,020 INFO [RS:2;jenkins-hbase9:43229] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,43229,1689339994795/jenkins-hbase9.apache.org%2C43229%2C1689339994795.1689339994991 2023-07-14 13:06:35,021 DEBUG [RS:2;jenkins-hbase9:43229] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44567,DS-d1fe6f1d-a2c6-4e10-b2ab-1a91df578e1c,DISK], DatanodeInfoWithStorage[127.0.0.1:45715,DS-d0e9d4fb-a902-4f72-9d50-25e21bea1880,DISK]] 2023-07-14 13:06:37,437 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-14 13:06:37,534 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-14 13:06:37,535 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-14 13:06:39,340 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-14 13:06:39,340 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-14 13:06:49,904 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease 383419177987211283 expired clientIPAndPort=172.31.2.10:44168, userName=jenkins, regionInfo=t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:51,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33743] regionserver.RSRpcServices(3094): Client tried to access missing scanner 383419177987211283 2023-07-14 13:06:51,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33743] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.2.10:44168 deadline: 1689340071904, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '383419177987211283'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-14 13:06:51,906 DEBUG [Listener at localhost/40033] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '383419177987211283'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '383419177987211283'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '383419177987211283'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-14 13:06:51,910 DEBUG [Listener at localhost/40033] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d07ef26 to 127.0.0.1:60854 2023-07-14 13:06:51,910 DEBUG [Listener at localhost/40033] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 13:06:51,910 INFO [Listener at localhost/40033] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-14 13:06:51,925 INFO [Listener at localhost/40033] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=294 (was 270) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60854@0x4d07ef26-SendThread(127.0.0.1:60854) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=43229 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:34955 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43229 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60854@0x75643ef5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/732401463.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf-prefix:jenkins-hbase9.apache.org,43229,1689339994795 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43229 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60854@0x75643ef5-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43229 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=43229 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1698479769_17 at /127.0.0.1:54834 [Receiving block BP-1857557694-172.31.2.10-1689339986089:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/cluster_b78f3397-94fd-c6dd-412f-27eb5739866b/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase9:43229-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60854@0x75643ef5-SendThread(127.0.0.1:60854) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: jenkins-hbase9:43229Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60854@0x4d07ef26 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/732401463.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43229 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-1857557694-172.31.2.10-1689339986089:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=43229 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/cluster_b78f3397-94fd-c6dd-412f-27eb5739866b/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=43229 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60854@0x4d07ef26-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-1857557694-172.31.2.10-1689339986089:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase9:43229 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1698479769_17 at /127.0.0.1:50160 [Receiving block BP-1857557694-172.31.2.10-1689339986089:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43229 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=43229 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) - Thread LEAK? -, OpenFileDescriptor=571 (was 567) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=106 (was 136), ProcessCount=174 (was 174), AvailableMemoryMB=11360 (was 11434) 2023-07-14 13:06:51,938 INFO [Listener at localhost/40033] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=294, OpenFileDescriptor=571, MaxFileDescriptor=60000, SystemLoadAverage=106, ProcessCount=174, AvailableMemoryMB=11359 2023-07-14 13:06:51,938 INFO [Listener at localhost/40033] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase9:38891 2023-07-14 13:06:51,938 INFO [Listener at localhost/40033] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase9.apache.org,38891,1689339990374 2023-07-14 13:06:51,938 INFO [Listener at localhost/40033] client.TestScannerTimeout(104): START************ test2772 2023-07-14 13:06:51,941 DEBUG [Listener at localhost/40033] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-14 13:06:51,941 DEBUG [Listener at localhost/40033] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:51,942 DEBUG [Listener at localhost/40033] zookeeper.ReadOnlyZKClient(139): Connect 0x0bee89e8 to 127.0.0.1:60854 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 13:06:51,947 DEBUG [Listener at localhost/40033] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7fffe7ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 13:06:51,947 ERROR [Listener at localhost/40033] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase9.apache.org,33743,1689339990331: die! ***** 2023-07-14 13:06:51,948 ERROR [Listener at localhost/40033] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-14 13:06:51,948 DEBUG [Listener at localhost/40033] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-14 13:06:51,949 DEBUG [Listener at localhost/40033] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-14 13:06:51,954 DEBUG [Listener at localhost/40033] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-14 13:06:51,956 DEBUG [Listener at localhost/40033] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-14 13:06:51,974 INFO [Listener at localhost/40033] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1152385024, "init": 513802240, "max": 2051014656, "used": 219692128 }, "NonHeapMemoryUsage": { "committed": 122183680, "init": 2555904, "max": -1, "used": 119708240 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase9.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 3, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 11956, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 61, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 61, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 61, "TotalCallTime_min": 1, "TotalCallTime_max": 1, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 61, "ResponseSize_min": 196, "ResponseSize_max": 678, "ResponseSize_mean": 437, "ResponseSize_25th_percentile": 196, "ResponseSize_median": 196, "ResponseSize_75th_percentile": 675, "ResponseSize_90th_percentile": 677, "ResponseSize_95th_percentile": 677, "ResponseSize_98th_percentile": 677, "ResponseSize_99th_percentile": 677, "ResponseSize_99.9th_percentile": 677, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 4, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 61, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16660 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase9.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:60854", "tag.serverName": "jenkins-hbase9.apache.org,38891,1689339990374", "tag.clusterId": "88dab552-822c-46d4-9648-1529c38da56c", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase9.apache.org", "regionCount": 0, "storeCount": 0, "hlogFileCount": 1, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 0, "avgStoreFileAge": 0, "numReferenceFiles": 0, "regionServerStartTime": 1689339990374, "averageRegionSize": 0, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 546, "localBytesRead": 546, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 1640, "ByteBuffAllocatorPoolAllocationBytes": 66560, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 11, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 0, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 1, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 10, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 1, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 10, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 0, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 0, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 0, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 0, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 1, "ScanSize_min": 0, "ScanSize_max": 0, "ScanSize_mean": 0, "ScanSize_25th_percentile": 0, "ScanSize_median": 0, "ScanSize_75th_percentile": 0, "ScanSize_90th_percentile": 0, "ScanSize_95th_percentile": 0, "ScanSize_98th_percentile": 0, "ScanSize_99th_percentile": 0, "ScanSize_99.9th_percentile": 0, "slowGetCount": 0, "flushedMemstoreBytes": 0, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 0, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.641128781747236, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 24, "ServerWriteQueryPerSecond_mean_rate": 1.1738347786057965, "ServerWriteQueryPerSecond_1min_rate": 3.738243758742744, "ServerWriteQueryPerSecond_5min_rate": 4.565901237603426, "ServerWriteQueryPerSecond_15min_rate": 4.720662978343764 } ] } 2023-07-14 13:06:51,975 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45203] master.MasterRpcServices(612): jenkins-hbase9.apache.org,33743,1689339990331 reported a fatal error: ***** ABORTING region server jenkins-hbase9.apache.org,33743,1689339990331: die! ***** 2023-07-14 13:06:51,976 INFO [Listener at localhost/40033] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,33743,1689339990331' ***** 2023-07-14 13:06:51,977 INFO [Listener at localhost/40033] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-14 13:06:51,977 INFO [RS:0;jenkins-hbase9:33743] regionserver.HeapMemoryManager(220): Stopping 2023-07-14 13:06:51,977 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-14 13:06:51,977 INFO [RS:0;jenkins-hbase9:33743] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-14 13:06:51,978 INFO [RS:0;jenkins-hbase9:33743] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-14 13:06:51,978 INFO [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(3305): Received CLOSE for 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:51,978 INFO [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(3305): Received CLOSE for 96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:51,978 INFO [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(1141): aborting server jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:51,979 DEBUG [RS:0;jenkins-hbase9:33743] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c2bb844 to 127.0.0.1:60854 2023-07-14 13:06:51,979 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 148d9abb69f2f1f614322f28262c0164, disabling compactions & flushes 2023-07-14 13:06:51,979 DEBUG [RS:0;jenkins-hbase9:33743] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 13:06:51,979 INFO [RS:0;jenkins-hbase9:33743] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-14 13:06:51,979 INFO [RS:0;jenkins-hbase9:33743] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-14 13:06:51,979 INFO [RS:0;jenkins-hbase9:33743] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-14 13:06:51,979 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:51,979 INFO [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-14 13:06:51,979 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:51,979 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689339992857.148d9abb69f2f1f614322f28262c0164. after waiting 0 ms 2023-07-14 13:06:51,979 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:51,980 INFO [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-14 13:06:51,980 DEBUG [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(1478): Online Regions={148d9abb69f2f1f614322f28262c0164=t,,1689339992857.148d9abb69f2f1f614322f28262c0164., 1588230740=hbase:meta,,1.1588230740, 96683f9c76c93ef3cd5418f55c4f230c=hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c.} 2023-07-14 13:06:51,981 DEBUG [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(1504): Waiting on 148d9abb69f2f1f614322f28262c0164, 1588230740, 96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:51,981 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-14 13:06:51,981 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-14 13:06:51,981 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-14 13:06:51,981 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-14 13:06:51,981 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-14 13:06:51,983 DEBUG [hconnection-0x4ea0a8b5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 13:06:51,983 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1825): Memstore data size is 3228 in region hbase:meta,,1.1588230740 2023-07-14 13:06:51,984 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-14 13:06:51,986 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-14 13:06:51,986 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-14 13:06:51,987 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-14 13:06:51,989 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:51,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 148d9abb69f2f1f614322f28262c0164: 2023-07-14 13:06:51,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:51,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 96683f9c76c93ef3cd5418f55c4f230c, disabling compactions & flushes 2023-07-14 13:06:51,989 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:51,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:51,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. after waiting 0 ms 2023-07-14 13:06:51,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:51,990 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:51,990 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:53790, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 13:06:51,990 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:51,990 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 96683f9c76c93ef3cd5418f55c4f230c: 2023-07-14 13:06:51,991 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:51,991 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33743] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,33743,1689339990331 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 13:06:51,991 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33743] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.2.10:53790 deadline: 1689340071991, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,33743,1689339990331 aborting 2023-07-14 13:06:52,076 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-14 13:06:52,094 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33743] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,33743,1689339990331 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 13:06:52,094 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33743] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.2.10:53790 deadline: 1689340072094, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase9.apache.org,33743,1689339990331 aborting 2023-07-14 13:06:52,181 INFO [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,33743,1689339990331; all regions closed. 2023-07-14 13:06:52,195 DEBUG [RS:0;jenkins-hbase9:33743] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 13:06:52,195 INFO [RS:0;jenkins-hbase9:33743] regionserver.LeaseManager(133): Closed leases 2023-07-14 13:06:52,195 INFO [RS:0;jenkins-hbase9:33743] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-14 13:06:52,196 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-14 13:06:52,196 INFO [RS:0;jenkins-hbase9:33743] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:33743 2023-07-14 13:06:52,200 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:52,200 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:43229-0x10163e34d0a0008, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33743,1689339990331 2023-07-14 13:06:52,200 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 13:06:52,200 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 13:06:52,200 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:43229-0x10163e34d0a0008, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 13:06:52,202 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,33743,1689339990331] 2023-07-14 13:06:52,202 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,33743,1689339990331; numProcessing=1 2023-07-14 13:06:52,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43229-0x10163e34d0a0008, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:52,202 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase9.apache.org,33743,1689339990331 znode expired, triggering replicatorRemoved event 2023-07-14 13:06:52,204 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,33743,1689339990331 already deleted, retry=false 2023-07-14 13:06:52,204 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase9.apache.org,33743,1689339990331 on jenkins-hbase9.apache.org,45203,1689339989183 2023-07-14 13:06:52,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:43229-0x10163e34d0a0008, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:52,205 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=17, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase9.apache.org,33743,1689339990331, splitWal=true, meta=true 2023-07-14 13:06:52,206 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=17 for jenkins-hbase9.apache.org,33743,1689339990331 (carryingMeta=true) jenkins-hbase9.apache.org,33743,1689339990331/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@4be4d882[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-14 13:06:52,207 INFO [PEWorker-2] procedure.ServerCrashProcedure(161): Start pid=17, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,33743,1689339990331, splitWal=true, meta=true 2023-07-14 13:06:52,208 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,33743,1689339990331, splitWal=true, meta=true, isMeta: true 2023-07-14 13:06:52,211 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting 2023-07-14 13:06:52,213 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase9.apache.org,33743,1689339990331 WAL count=1, meta=true 2023-07-14 13:06:52,214 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta}] 2023-07-14 13:06:52,216 DEBUG [PEWorker-1] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:52,217 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta, worker=jenkins-hbase9.apache.org,43229,1689339994795}] 2023-07-14 13:06:52,297 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:33743 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:33743 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 13:06:52,298 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:33743 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:33743 2023-07-14 13:06:52,302 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 13:06:52,302 INFO [RS:0;jenkins-hbase9:33743] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,33743,1689339990331; zookeeper connection closed. 2023-07-14 13:06:52,302 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:33743-0x10163e34d0a0001, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 13:06:52,302 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@10d5c91b] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@10d5c91b 2023-07-14 13:06:52,370 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:52,370 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-14 13:06:52,371 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:34284, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-14 13:06:52,372 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43229] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=19 2023-07-14 13:06:52,388 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta, size=4.0 K (4123bytes) 2023-07-14 13:06:52,388 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta 2023-07-14 13:06:52,389 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta after 1ms 2023-07-14 13:06:52,393 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(311): Open hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta took 5ms 2023-07-14 13:06:52,397 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-14 13:06:52,399 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta so closing down 2023-07-14 13:06:52,399 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-14 13:06:52,402 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta.temp 2023-07-14 13:06:52,403 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-14 13:06:52,412 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) 2023-07-14 13:06:52,414 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta.temp to hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-14 13:06:52,414 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(423): Processed 13 edits across 1 Regions in 21 ms; skipped=1; WAL=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta, size=4.0 K, length=4123, corrupted=false, cancelled=false 2023-07-14 13:06:52,415 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta, journal: Splitting hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta, size=4.0 K (4123bytes) at 1689340012388Finishing writing output for hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta so closing down at 1689340012399 (+11 ms)Creating recovered edits writer path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta.temp at 1689340012402 (+3 ms)3 split writer threads finished at 1689340012403 (+1 ms)Closed recovered edits writer path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) at 1689340012412 (+9 ms)Rename recovered edits hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta.temp to hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 at 1689340012414 (+2 ms)Processed 13 edits across 1 Regions in 21 ms; skipped=1; WAL=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta, size=4.0 K, length=4123, corrupted=false, cancelled=false at 1689340012414 2023-07-14 13:06:52,415 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta 2023-07-14 13:06:52,415 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=19 2023-07-14 13:06:52,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster(3641): Remote procedure done, pid=19 2023-07-14 13:06:52,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] wal.WALSplitUtil(141): Moved hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta to hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/oldWALs 2023-07-14 13:06:52,422 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=18 2023-07-14 13:06:52,423 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta, worker=jenkins-hbase9.apache.org,43229,1689339994795 in 203 msec 2023-07-14 13:06:52,424 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:52,427 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-14 13:06:52,427 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase9.apache.org%2C33743%2C1689339990331.meta.1689339991849.meta, worker=jenkins-hbase9.apache.org,43229,1689339994795 in 211 msec 2023-07-14 13:06:52,429 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting dir is empty, no logs to split. 2023-07-14 13:06:52,429 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase9.apache.org,33743,1689339990331 WAL count=0, meta=true 2023-07-14 13:06:52,429 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase9.apache.org,33743,1689339990331 WAL splitting is done? wals=0, meta=true 2023-07-14 13:06:52,430 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-14 13:06:52,431 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-14 13:06:52,432 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 13:06:52,582 DEBUG [jenkins-hbase9:45203] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-14 13:06:52,583 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,43229,1689339994795, state=OPENING 2023-07-14 13:06:52,585 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-14 13:06:52,585 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 13:06:52,585 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,43229,1689339994795}] 2023-07-14 13:06:52,599 WARN [ReadOnlyZKClient-127.0.0.1:60854@0x0bee89e8] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-14 13:06:52,600 DEBUG [hconnection-0x4ea0a8b5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 13:06:52,603 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:34288, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 13:06:52,603 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43229] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 111 connection: 172.31.2.10:34288 deadline: 1689340072603, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:52,743 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-14 13:06:52,743 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 13:06:52,746 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C43229%2C1689339994795.meta, suffix=.meta, logDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,43229,1689339994795, archiveDir=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/oldWALs, maxLogs=32 2023-07-14 13:06:52,766 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44567,DS-d1fe6f1d-a2c6-4e10-b2ab-1a91df578e1c,DISK] 2023-07-14 13:06:52,766 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45715,DS-d0e9d4fb-a902-4f72-9d50-25e21bea1880,DISK] 2023-07-14 13:06:52,768 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,43229,1689339994795/jenkins-hbase9.apache.org%2C43229%2C1689339994795.meta.1689340012748.meta 2023-07-14 13:06:52,771 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44567,DS-d1fe6f1d-a2c6-4e10-b2ab-1a91df578e1c,DISK], DatanodeInfoWithStorage[127.0.0.1:45715,DS-d0e9d4fb-a902-4f72-9d50-25e21bea1880,DISK]] 2023-07-14 13:06:52,771 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-14 13:06:52,771 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-14 13:06:52,772 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-14 13:06:52,772 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-14 13:06:52,772 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-14 13:06:52,772 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 13:06:52,772 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-14 13:06:52,773 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-14 13:06:52,774 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-14 13:06:52,776 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/info 2023-07-14 13:06:52,776 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/info 2023-07-14 13:06:52,776 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-14 13:06:52,778 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 13:06:52,778 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-14 13:06:52,780 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/rep_barrier 2023-07-14 13:06:52,780 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/rep_barrier 2023-07-14 13:06:52,780 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-14 13:06:52,781 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 13:06:52,781 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-14 13:06:52,782 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/table 2023-07-14 13:06:52,782 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/table 2023-07-14 13:06:52,783 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-14 13:06:52,783 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 13:06:52,784 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740 2023-07-14 13:06:52,787 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740 2023-07-14 13:06:52,788 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-14 13:06:52,794 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5464): Applied 32, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=15, path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-14 13:06:52,794 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=3.15 KB heapSize=6.27 KB 2023-07-14 13:06:52,814 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.00 KB at sequenceid=15 (bloomFilter=false), to=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/.tmp/info/10e13a7bb976427986033dadc5fd8a09 2023-07-14 13:06:52,834 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=15 (bloomFilter=false), to=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/.tmp/table/588d0e105674480ea39d63247cac1805 2023-07-14 13:06:52,842 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/.tmp/info/10e13a7bb976427986033dadc5fd8a09 as hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/info/10e13a7bb976427986033dadc5fd8a09 2023-07-14 13:06:52,851 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/info/10e13a7bb976427986033dadc5fd8a09, entries=28, sequenceid=15, filesize=7.7 K 2023-07-14 13:06:52,852 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/.tmp/table/588d0e105674480ea39d63247cac1805 as hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/table/588d0e105674480ea39d63247cac1805 2023-07-14 13:06:52,859 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/table/588d0e105674480ea39d63247cac1805, entries=4, sequenceid=15, filesize=4.7 K 2023-07-14 13:06:52,859 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.15 KB/3228, heapSize ~5.98 KB/6128, currentSize=0 B/0 for 1588230740 in 65ms, sequenceid=15, compaction requested=false; wal=null 2023-07-14 13:06:52,861 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-14 13:06:52,864 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-14 13:06:52,865 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-14 13:06:52,868 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=1 2023-07-14 13:06:52,869 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74689490, jitterRate=0.11296012997627258}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-14 13:06:52,869 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-14 13:06:52,870 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=21, masterSystemTime=1689340012737 2023-07-14 13:06:52,872 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-14 13:06:52,872 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-14 13:06:52,873 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,43229,1689339994795, state=OPEN 2023-07-14 13:06:52,874 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-14 13:06:52,874 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 13:06:52,877 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-14 13:06:52,877 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,43229,1689339994795 in 289 msec 2023-07-14 13:06:52,879 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=17 2023-07-14 13:06:52,880 INFO [PEWorker-1] procedure.ServerCrashProcedure(199): jenkins-hbase9.apache.org,33743,1689339990331 had 3 regions 2023-07-14 13:06:52,880 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 447 msec 2023-07-14 13:06:52,880 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase9.apache.org,33743,1689339990331, splitWal=true, meta=true, isMeta: false 2023-07-14 13:06:52,882 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase9.apache.org,33743,1689339990331 WAL count=1, meta=false 2023-07-14 13:06:52,883 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695}] 2023-07-14 13:06:52,884 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:52,886 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695, worker=jenkins-hbase9.apache.org,43229,1689339994795}] 2023-07-14 13:06:53,038 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43229] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=23 2023-07-14 13:06:53,052 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695, size=899 (899bytes) 2023-07-14 13:06:53,052 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695 2023-07-14 13:06:53,052 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695 after 0ms 2023-07-14 13:06:53,056 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(311): Open hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695 took 5ms 2023-07-14 13:06:53,059 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(346): Last flushed sequenceid for 96683f9c76c93ef3cd5418f55c4f230c: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-14 13:06:53,062 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(346): Last flushed sequenceid for 148d9abb69f2f1f614322f28262c0164: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-14 13:06:53,062 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695 so closing down 2023-07-14 13:06:53,062 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-14 13:06:53,064 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695.temp 2023-07-14 13:06:53,064 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-14 13:06:53,073 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-14 13:06:53,075 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695.temp to hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/recovered.edits/0000000000000000005 2023-07-14 13:06:53,075 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(423): Processed 4 edits across 1 Regions in 18 ms; skipped=2; WAL=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695, size=899, length=899, corrupted=false, cancelled=false 2023-07-14 13:06:53,075 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695, journal: Splitting hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695, size=899 (899bytes) at 1689340013052Finishing writing output for hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695 so closing down at 1689340013062 (+10 ms)Creating recovered edits writer path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695.temp at 1689340013064 (+2 ms)3 split writer threads finished at 1689340013064Closed recovered edits writer path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689340013073 (+9 ms)Rename recovered edits hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/recovered.edits/0000000000000000004-jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695.temp to hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/recovered.edits/0000000000000000005 at 1689340013075 (+2 ms)Processed 4 edits across 1 Regions in 18 ms; skipped=2; WAL=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695, size=899, length=899, corrupted=false, cancelled=false at 1689340013075 2023-07-14 13:06:53,075 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695 2023-07-14 13:06:53,075 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=23 2023-07-14 13:06:53,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster(3641): Remote procedure done, pid=23 2023-07-14 13:06:53,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] wal.WALSplitUtil(141): Moved hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting/jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695 to hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/oldWALs 2023-07-14 13:06:53,082 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-14 13:06:53,082 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695, worker=jenkins-hbase9.apache.org,43229,1689339994795 in 194 msec 2023-07-14 13:06:53,083 DEBUG [PEWorker-2] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:53,086 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=17 2023-07-14 13:06:53,086 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase9.apache.org%2C33743%2C1689339990331.1689339991695, worker=jenkins-hbase9.apache.org,43229,1689339994795 in 201 msec 2023-07-14 13:06:53,087 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/WALs/jenkins-hbase9.apache.org,33743,1689339990331-splitting dir is empty, no logs to split. 2023-07-14 13:06:53,087 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase9.apache.org,33743,1689339990331 WAL count=0, meta=false 2023-07-14 13:06:53,087 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase9.apache.org,33743,1689339990331 WAL splitting is done? wals=0, meta=false 2023-07-14 13:06:53,089 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=96683f9c76c93ef3cd5418f55c4f230c, ASSIGN}, {pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=148d9abb69f2f1f614322f28262c0164, ASSIGN}] 2023-07-14 13:06:53,090 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=96683f9c76c93ef3cd5418f55c4f230c, ASSIGN 2023-07-14 13:06:53,090 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=148d9abb69f2f1f614322f28262c0164, ASSIGN 2023-07-14 13:06:53,091 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=148d9abb69f2f1f614322f28262c0164, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 13:06:53,091 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=96683f9c76c93ef3cd5418f55c4f230c, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 13:06:53,092 DEBUG [jenkins-hbase9:45203] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-14 13:06:53,092 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=96683f9c76c93ef3cd5418f55c4f230c, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:53,092 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=148d9abb69f2f1f614322f28262c0164, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:53,093 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689340013092"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689340013092"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689340013092"}]},"ts":"1689340013092"} 2023-07-14 13:06:53,093 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689339992857.148d9abb69f2f1f614322f28262c0164.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689340013092"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689340013092"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689340013092"}]},"ts":"1689340013092"} 2023-07-14 13:06:53,094 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:33743 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:33743 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 13:06:53,095 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:33743 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:33743 2023-07-14 13:06:53,116 DEBUG [hconnection-0x4ea0a8b5-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase9.apache.org/172.31.2.10:33743 this server is in the failed servers list 2023-07-14 13:06:53,197 DEBUG [PEWorker-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 13:06:53,198 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:34292, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 13:06:53,200 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; OpenRegionProcedure 148d9abb69f2f1f614322f28262c0164, server=jenkins-hbase9.apache.org,43229,1689339994795}] 2023-07-14 13:06:53,201 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=24, state=RUNNABLE; OpenRegionProcedure 96683f9c76c93ef3cd5418f55c4f230c, server=jenkins-hbase9.apache.org,43229,1689339994795}] 2023-07-14 13:06:53,224 DEBUG [hconnection-0x4ea0a8b5-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase9.apache.org/172.31.2.10:33743 this server is in the failed servers list 2023-07-14 13:06:53,357 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:53,357 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 96683f9c76c93ef3cd5418f55c4f230c, NAME => 'hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c.', STARTKEY => '', ENDKEY => ''} 2023-07-14 13:06:53,358 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:53,358 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 13:06:53,358 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:53,358 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:53,360 INFO [StoreOpener-96683f9c76c93ef3cd5418f55c4f230c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:53,361 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:53,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 148d9abb69f2f1f614322f28262c0164, NAME => 't,,1689339992857.148d9abb69f2f1f614322f28262c0164.', STARTKEY => '', ENDKEY => ''} 2023-07-14 13:06:53,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:53,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated t,,1689339992857.148d9abb69f2f1f614322f28262c0164.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 13:06:53,361 DEBUG [StoreOpener-96683f9c76c93ef3cd5418f55c4f230c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/info 2023-07-14 13:06:53,361 DEBUG [StoreOpener-96683f9c76c93ef3cd5418f55c4f230c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/info 2023-07-14 13:06:53,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:53,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:53,362 INFO [StoreOpener-96683f9c76c93ef3cd5418f55c4f230c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96683f9c76c93ef3cd5418f55c4f230c columnFamilyName info 2023-07-14 13:06:53,363 INFO [StoreOpener-96683f9c76c93ef3cd5418f55c4f230c-1] regionserver.HStore(310): Store=96683f9c76c93ef3cd5418f55c4f230c/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 13:06:53,363 INFO [StoreOpener-148d9abb69f2f1f614322f28262c0164-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:53,364 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:53,364 DEBUG [StoreOpener-148d9abb69f2f1f614322f28262c0164-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/f 2023-07-14 13:06:53,365 DEBUG [StoreOpener-148d9abb69f2f1f614322f28262c0164-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/f 2023-07-14 13:06:53,365 INFO [StoreOpener-148d9abb69f2f1f614322f28262c0164-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 148d9abb69f2f1f614322f28262c0164 columnFamilyName f 2023-07-14 13:06:53,367 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:53,368 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/recovered.edits/0000000000000000005 2023-07-14 13:06:53,372 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/recovered.edits/0000000000000000005 2023-07-14 13:06:53,374 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 96683f9c76c93ef3cd5418f55c4f230c 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-14 13:06:53,376 DEBUG [StoreOpener-148d9abb69f2f1f614322f28262c0164-1] regionserver.HStore(539): loaded hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/f/f5dc6056b7ed4735861f59e27681b803 2023-07-14 13:06:53,377 INFO [StoreOpener-148d9abb69f2f1f614322f28262c0164-1] regionserver.HStore(310): Store=148d9abb69f2f1f614322f28262c0164/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 13:06:53,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:53,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:53,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:53,386 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-14 13:06:53,390 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 148d9abb69f2f1f614322f28262c0164; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74816828, jitterRate=0.11485761404037476}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 13:06:53,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 148d9abb69f2f1f614322f28262c0164: 2023-07-14 13:06:53,392 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689339992857.148d9abb69f2f1f614322f28262c0164., pid=26, masterSystemTime=1689340013353 2023-07-14 13:06:53,396 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/.tmp/info/eb0ef9a080a9436784c6ccd65417e982 2023-07-14 13:06:53,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:53,397 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:53,398 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=148d9abb69f2f1f614322f28262c0164, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:53,398 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689339992857.148d9abb69f2f1f614322f28262c0164.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689340013397"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689340013397"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689340013397"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689340013397"}]},"ts":"1689340013397"} 2023-07-14 13:06:53,403 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-07-14 13:06:53,403 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; OpenRegionProcedure 148d9abb69f2f1f614322f28262c0164, server=jenkins-hbase9.apache.org,43229,1689339994795 in 200 msec 2023-07-14 13:06:53,405 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=t, region=148d9abb69f2f1f614322f28262c0164, ASSIGN in 314 msec 2023-07-14 13:06:53,406 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/.tmp/info/eb0ef9a080a9436784c6ccd65417e982 as hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/info/eb0ef9a080a9436784c6ccd65417e982 2023-07-14 13:06:53,413 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/info/eb0ef9a080a9436784c6ccd65417e982, entries=2, sequenceid=5, filesize=4.8 K 2023-07-14 13:06:53,414 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 96683f9c76c93ef3cd5418f55c4f230c in 40ms, sequenceid=5, compaction requested=false; wal=null 2023-07-14 13:06:53,415 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/recovered.edits/0000000000000000005 2023-07-14 13:06:53,419 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:53,421 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-14 13:06:53,422 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 96683f9c76c93ef3cd5418f55c4f230c; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59488855, jitterRate=-0.11354698240756989}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 13:06:53,422 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 96683f9c76c93ef3cd5418f55c4f230c: 2023-07-14 13:06:53,423 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c., pid=27, masterSystemTime=1689340013353 2023-07-14 13:06:53,425 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:53,425 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:53,426 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=96683f9c76c93ef3cd5418f55c4f230c, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:53,426 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689340013426"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689340013426"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689340013426"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689340013426"}]},"ts":"1689340013426"} 2023-07-14 13:06:53,431 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=24 2023-07-14 13:06:53,431 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=24, state=SUCCESS; OpenRegionProcedure 96683f9c76c93ef3cd5418f55c4f230c, server=jenkins-hbase9.apache.org,43229,1689339994795 in 227 msec 2023-07-14 13:06:53,434 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=17 2023-07-14 13:06:53,435 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase9.apache.org,33743,1689339990331 after splitting done 2023-07-14 13:06:53,435 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=96683f9c76c93ef3cd5418f55c4f230c, ASSIGN in 342 msec 2023-07-14 13:06:53,435 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase9.apache.org,33743,1689339990331 from processing; numProcessing=0 2023-07-14 13:06:53,437 DEBUG [Listener at localhost/40033] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bee89e8 to 127.0.0.1:60854 2023-07-14 13:06:53,437 DEBUG [Listener at localhost/40033] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 13:06:53,438 INFO [Listener at localhost/40033] client.TestScannerTimeout(124): END ************ test2772 2023-07-14 13:06:53,438 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, state=SUCCESS; ServerCrashProcedure jenkins-hbase9.apache.org,33743,1689339990331, splitWal=true, meta=true in 1.2310 sec 2023-07-14 13:06:53,452 INFO [Listener at localhost/40033] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=275 (was 294), OpenFileDescriptor=586 (was 571) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=97 (was 106), ProcessCount=174 (was 174), AvailableMemoryMB=11359 (was 11359) 2023-07-14 13:06:53,453 INFO [Listener at localhost/40033] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-14 13:06:53,453 INFO [Listener at localhost/40033] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-14 13:06:53,453 DEBUG [Listener at localhost/40033] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38241035 to 127.0.0.1:60854 2023-07-14 13:06:53,453 DEBUG [Listener at localhost/40033] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 13:06:53,454 DEBUG [Listener at localhost/40033] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-14 13:06:53,454 DEBUG [Listener at localhost/40033] util.JVMClusterUtil(257): Found active master hash=392371017, stopped=false 2023-07-14 13:06:53,454 INFO [Listener at localhost/40033] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase9.apache.org,45203,1689339989183 2023-07-14 13:06:53,460 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:43229-0x10163e34d0a0008, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-14 13:06:53,460 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-14 13:06:53,460 INFO [Listener at localhost/40033] procedure2.ProcedureExecutor(629): Stopping 2023-07-14 13:06:53,460 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 13:06:53,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43229-0x10163e34d0a0008, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 13:06:53,460 DEBUG [Listener at localhost/40033] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2159cf25 to 127.0.0.1:60854 2023-07-14 13:06:53,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 13:06:53,460 DEBUG [Listener at localhost/40033] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 13:06:53,461 INFO [Listener at localhost/40033] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,43229,1689339994795' ***** 2023-07-14 13:06:53,461 INFO [Listener at localhost/40033] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-14 13:06:53,461 INFO [RS:2;jenkins-hbase9:43229] regionserver.HeapMemoryManager(220): Stopping 2023-07-14 13:06:53,461 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-14 13:06:53,461 INFO [RS:2;jenkins-hbase9:43229] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-14 13:06:53,461 INFO [RS:2;jenkins-hbase9:43229] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-14 13:06:53,461 INFO [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(3305): Received CLOSE for 148d9abb69f2f1f614322f28262c0164 2023-07-14 13:06:53,462 INFO [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(3305): Received CLOSE for 96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:53,462 INFO [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(1144): stopping server jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:53,462 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 148d9abb69f2f1f614322f28262c0164, disabling compactions & flushes 2023-07-14 13:06:53,462 DEBUG [RS:2;jenkins-hbase9:43229] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x75643ef5 to 127.0.0.1:60854 2023-07-14 13:06:53,462 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:53,462 DEBUG [RS:2;jenkins-hbase9:43229] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 13:06:53,462 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:53,462 INFO [RS:2;jenkins-hbase9:43229] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-14 13:06:53,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689339992857.148d9abb69f2f1f614322f28262c0164. after waiting 0 ms 2023-07-14 13:06:53,463 INFO [RS:2;jenkins-hbase9:43229] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-14 13:06:53,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:53,463 INFO [RS:2;jenkins-hbase9:43229] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-14 13:06:53,463 INFO [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-14 13:06:53,463 INFO [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-14 13:06:53,463 DEBUG [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(1478): Online Regions={148d9abb69f2f1f614322f28262c0164=t,,1689339992857.148d9abb69f2f1f614322f28262c0164., 1588230740=hbase:meta,,1.1588230740, 96683f9c76c93ef3cd5418f55c4f230c=hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c.} 2023-07-14 13:06:53,463 DEBUG [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(1504): Waiting on 148d9abb69f2f1f614322f28262c0164, 1588230740, 96683f9c76c93ef3cd5418f55c4f230c 2023-07-14 13:06:53,464 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-14 13:06:53,464 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-14 13:06:53,464 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-14 13:06:53,464 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-14 13:06:53,464 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-14 13:06:53,465 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-14 13:06:53,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/default/t/148d9abb69f2f1f614322f28262c0164/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-14 13:06:53,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:53,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 148d9abb69f2f1f614322f28262c0164: 2023-07-14 13:06:53,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed t,,1689339992857.148d9abb69f2f1f614322f28262c0164. 2023-07-14 13:06:53,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 96683f9c76c93ef3cd5418f55c4f230c, disabling compactions & flushes 2023-07-14 13:06:53,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:53,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:53,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. after waiting 0 ms 2023-07-14 13:06:53,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:53,481 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/namespace/96683f9c76c93ef3cd5418f55c4f230c/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=5 2023-07-14 13:06:53,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:53,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 96683f9c76c93ef3cd5418f55c4f230c: 2023-07-14 13:06:53,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689339992114.96683f9c76c93ef3cd5418f55c4f230c. 2023-07-14 13:06:53,497 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=22 (bloomFilter=false), to=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/.tmp/info/0f66b8d1210b4ac9bcf7f65c9c12bccc 2023-07-14 13:06:53,508 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/.tmp/info/0f66b8d1210b4ac9bcf7f65c9c12bccc as hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/info/0f66b8d1210b4ac9bcf7f65c9c12bccc 2023-07-14 13:06:53,517 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/info/0f66b8d1210b4ac9bcf7f65c9c12bccc, entries=16, sequenceid=22, filesize=6.4 K 2023-07-14 13:06:53,518 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 53ms, sequenceid=22, compaction requested=false 2023-07-14 13:06:53,538 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/data/hbase/meta/1588230740/recovered.edits/25.seqid, newMaxSeqId=25, maxSeqId=15 2023-07-14 13:06:53,540 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-14 13:06:53,540 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-14 13:06:53,540 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-14 13:06:53,540 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-14 13:06:53,664 INFO [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,43229,1689339994795; all regions closed. 2023-07-14 13:06:53,669 DEBUG [RS:2;jenkins-hbase9:43229] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/oldWALs 2023-07-14 13:06:53,670 INFO [RS:2;jenkins-hbase9:43229] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C43229%2C1689339994795.meta:.meta(num 1689340012748) 2023-07-14 13:06:53,674 DEBUG [RS:2;jenkins-hbase9:43229] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/oldWALs 2023-07-14 13:06:53,674 INFO [RS:2;jenkins-hbase9:43229] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C43229%2C1689339994795:(num 1689339994991) 2023-07-14 13:06:53,674 DEBUG [RS:2;jenkins-hbase9:43229] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 13:06:53,674 INFO [RS:2;jenkins-hbase9:43229] regionserver.LeaseManager(133): Closed leases 2023-07-14 13:06:53,675 INFO [RS:2;jenkins-hbase9:43229] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-14 13:06:53,675 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-14 13:06:53,676 INFO [RS:2;jenkins-hbase9:43229] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:43229 2023-07-14 13:06:53,677 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:43229-0x10163e34d0a0008, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,43229,1689339994795 2023-07-14 13:06:53,677 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 13:06:53,678 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:43229-0x10163e34d0a0008, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 13:06:53,679 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,43229,1689339994795] 2023-07-14 13:06:53,679 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,43229,1689339994795; numProcessing=1 2023-07-14 13:06:53,681 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,43229,1689339994795 already deleted, retry=false 2023-07-14 13:06:53,681 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,43229,1689339994795 expired; onlineServers=0 2023-07-14 13:06:53,681 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,45203,1689339989183' ***** 2023-07-14 13:06:53,681 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-14 13:06:53,682 DEBUG [M:0;jenkins-hbase9:45203] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@565e1172, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-14 13:06:53,682 INFO [M:0;jenkins-hbase9:45203] regionserver.HRegionServer(1144): stopping server jenkins-hbase9.apache.org,45203,1689339989183 2023-07-14 13:06:53,682 INFO [M:0;jenkins-hbase9:45203] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,45203,1689339989183; all regions closed. 2023-07-14 13:06:53,682 DEBUG [M:0;jenkins-hbase9:45203] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 13:06:53,682 DEBUG [M:0;jenkins-hbase9:45203] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-14 13:06:53,682 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-14 13:06:53,682 DEBUG [M:0;jenkins-hbase9:45203] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-14 13:06:53,682 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689339991356] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689339991356,5,FailOnTimeoutGroup] 2023-07-14 13:06:53,683 INFO [M:0;jenkins-hbase9:45203] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-14 13:06:53,682 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689339991357] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689339991357,5,FailOnTimeoutGroup] 2023-07-14 13:06:53,683 INFO [M:0;jenkins-hbase9:45203] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-14 13:06:53,683 INFO [M:0;jenkins-hbase9:45203] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-07-14 13:06:53,683 DEBUG [M:0;jenkins-hbase9:45203] master.HMaster(1512): Stopping service threads 2023-07-14 13:06:53,683 INFO [M:0;jenkins-hbase9:45203] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-14 13:06:53,683 ERROR [M:0;jenkins-hbase9:45203] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (1124632911) connection to localhost/127.0.0.1:34955 from jenkins,5,PEWorkerGroup] 2023-07-14 13:06:53,684 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-14 13:06:53,684 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 13:06:53,684 INFO [M:0;jenkins-hbase9:45203] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-14 13:06:53,684 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-14 13:06:53,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-14 13:06:53,684 DEBUG [M:0;jenkins-hbase9:45203] zookeeper.ZKUtil(398): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-14 13:06:53,684 WARN [M:0;jenkins-hbase9:45203] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-14 13:06:53,684 INFO [M:0;jenkins-hbase9:45203] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-14 13:06:53,685 INFO [M:0;jenkins-hbase9:45203] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-14 13:06:53,685 DEBUG [M:0;jenkins-hbase9:45203] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-14 13:06:53,685 INFO [M:0;jenkins-hbase9:45203] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 13:06:53,685 DEBUG [M:0;jenkins-hbase9:45203] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 13:06:53,685 DEBUG [M:0;jenkins-hbase9:45203] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-14 13:06:53,685 DEBUG [M:0;jenkins-hbase9:45203] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 13:06:53,685 INFO [M:0;jenkins-hbase9:45203] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=90.77 KB heapSize=109.49 KB 2023-07-14 13:06:53,700 INFO [M:0;jenkins-hbase9:45203] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=90.77 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/eb2eb997a68d466ab75221325572779c 2023-07-14 13:06:53,707 DEBUG [M:0;jenkins-hbase9:45203] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/eb2eb997a68d466ab75221325572779c as hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/eb2eb997a68d466ab75221325572779c 2023-07-14 13:06:53,713 INFO [M:0;jenkins-hbase9:45203] regionserver.HStore(1080): Added hdfs://localhost:34955/user/jenkins/test-data/84751ec6-f9d4-0f9e-c116-6dcfbc21faaf/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/eb2eb997a68d466ab75221325572779c, entries=27, sequenceid=212, filesize=8.2 K 2023-07-14 13:06:53,714 INFO [M:0;jenkins-hbase9:45203] regionserver.HRegion(2948): Finished flush of dataSize ~90.77 KB/92953, heapSize ~109.48 KB/112104, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 29ms, sequenceid=212, compaction requested=false 2023-07-14 13:06:53,715 INFO [M:0;jenkins-hbase9:45203] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 13:06:53,715 DEBUG [M:0;jenkins-hbase9:45203] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-14 13:06:53,718 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-14 13:06:53,719 INFO [M:0;jenkins-hbase9:45203] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-14 13:06:53,719 INFO [M:0;jenkins-hbase9:45203] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:45203 2023-07-14 13:06:53,721 DEBUG [M:0;jenkins-hbase9:45203] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,45203,1689339989183 already deleted, retry=false 2023-07-14 13:06:53,866 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-14 13:06:53,942 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 13:06:53,942 INFO [M:0;jenkins-hbase9:45203] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,45203,1689339989183; zookeeper connection closed. 2023-07-14 13:06:53,943 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10163e34d0a0000, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 13:06:54,042 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:43229-0x10163e34d0a0008, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 13:06:54,043 INFO [RS:2;jenkins-hbase9:43229] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,43229,1689339994795; zookeeper connection closed. 2023-07-14 13:06:54,043 DEBUG [Listener at localhost/40033-EventThread] zookeeper.ZKWatcher(600): regionserver:43229-0x10163e34d0a0008, quorum=127.0.0.1:60854, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 13:06:54,043 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5f2d85d0] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5f2d85d0 2023-07-14 13:06:54,043 INFO [Listener at localhost/40033] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-14 13:06:54,044 WARN [Listener at localhost/40033] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-14 13:06:54,046 INFO [Listener at localhost/40033] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-14 13:06:54,153 WARN [BP-1857557694-172.31.2.10-1689339986089 heartbeating to localhost/127.0.0.1:34955] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-14 13:06:54,153 WARN [BP-1857557694-172.31.2.10-1689339986089 heartbeating to localhost/127.0.0.1:34955] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1857557694-172.31.2.10-1689339986089 (Datanode Uuid 3db6d24c-9770-4b94-bf1d-ce8bea397f3a) service to localhost/127.0.0.1:34955 2023-07-14 13:06:54,155 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/cluster_b78f3397-94fd-c6dd-412f-27eb5739866b/dfs/data/data3/current/BP-1857557694-172.31.2.10-1689339986089] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-14 13:06:54,155 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/cluster_b78f3397-94fd-c6dd-412f-27eb5739866b/dfs/data/data4/current/BP-1857557694-172.31.2.10-1689339986089] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-14 13:06:54,157 WARN [Listener at localhost/40033] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-14 13:06:54,159 INFO [Listener at localhost/40033] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-14 13:06:54,262 WARN [BP-1857557694-172.31.2.10-1689339986089 heartbeating to localhost/127.0.0.1:34955] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-14 13:06:54,262 WARN [BP-1857557694-172.31.2.10-1689339986089 heartbeating to localhost/127.0.0.1:34955] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1857557694-172.31.2.10-1689339986089 (Datanode Uuid 6cba91a1-18bb-431c-94a7-1b250f37eb9d) service to localhost/127.0.0.1:34955 2023-07-14 13:06:54,263 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/cluster_b78f3397-94fd-c6dd-412f-27eb5739866b/dfs/data/data1/current/BP-1857557694-172.31.2.10-1689339986089] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-14 13:06:54,263 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/867f768d-c586-935e-754b-02de9eaa124d/cluster_b78f3397-94fd-c6dd-412f-27eb5739866b/dfs/data/data2/current/BP-1857557694-172.31.2.10-1689339986089] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-14 13:06:54,298 INFO [Listener at localhost/40033] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-14 13:06:54,410 INFO [Listener at localhost/40033] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-14 13:06:54,444 INFO [Listener at localhost/40033] hbase.HBaseTestingUtility(1293): Minicluster is down