2023-07-15 19:05:20,209 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-15 19:05:20,447 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4 2023-07-15 19:05:20,460 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-15 19:05:20,460 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/cluster_a9497dea-7554-f5e1-575e-fef856a8e7af, deleteOnExit=true 2023-07-15 19:05:20,460 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-15 19:05:20,461 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/test.cache.data in system properties and HBase conf 2023-07-15 19:05:20,461 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/hadoop.tmp.dir in system properties and HBase conf 2023-07-15 19:05:20,461 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/hadoop.log.dir in system properties and HBase conf 2023-07-15 19:05:20,462 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-15 19:05:20,462 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-15 19:05:20,463 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-15 19:05:20,576 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-15 19:05:20,962 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-15 19:05:20,967 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-15 19:05:20,967 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-15 19:05:20,967 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-15 19:05:20,967 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-15 19:05:20,968 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-15 19:05:20,968 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-15 19:05:20,968 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-15 19:05:20,969 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-15 19:05:20,969 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-15 19:05:20,970 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/nfs.dump.dir in system properties and HBase conf 2023-07-15 19:05:20,970 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/java.io.tmpdir in system properties and HBase conf 2023-07-15 19:05:20,970 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-15 19:05:20,971 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-15 19:05:20,971 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-15 19:05:21,491 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-15 19:05:21,496 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-15 19:05:21,780 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-15 19:05:21,960 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-15 19:05:21,980 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-15 19:05:22,015 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-15 19:05:22,047 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/java.io.tmpdir/Jetty_localhost_33847_hdfs____m088hx/webapp 2023-07-15 19:05:22,202 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33847 2023-07-15 19:05:22,212 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-15 19:05:22,212 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-15 19:05:22,620 WARN [Listener at localhost/45815] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-15 19:05:22,689 WARN [Listener at localhost/45815] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-15 19:05:22,707 WARN [Listener at localhost/45815] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-15 19:05:22,715 INFO [Listener at localhost/45815] log.Slf4jLog(67): jetty-6.1.26 2023-07-15 19:05:22,719 INFO [Listener at localhost/45815] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/java.io.tmpdir/Jetty_localhost_42359_datanode____.efd29s/webapp 2023-07-15 19:05:22,821 INFO [Listener at localhost/45815] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42359 2023-07-15 19:05:23,141 WARN [Listener at localhost/40173] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-15 19:05:23,153 WARN [Listener at localhost/40173] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-15 19:05:23,157 WARN [Listener at localhost/40173] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-15 19:05:23,159 INFO [Listener at localhost/40173] log.Slf4jLog(67): jetty-6.1.26 2023-07-15 19:05:23,167 INFO [Listener at localhost/40173] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/java.io.tmpdir/Jetty_localhost_41873_datanode____.azq78i/webapp 2023-07-15 19:05:23,273 INFO [Listener at localhost/40173] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41873 2023-07-15 19:05:23,281 WARN [Listener at localhost/42557] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-15 19:05:23,604 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x13ae9aa296fd68ed: Processing first storage report for DS-0eeb57f1-6175-43f3-9611-c11b9699d2f2 from datanode c84b3dbf-c391-4f36-b298-3ef2e911d0d9 2023-07-15 19:05:23,605 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x13ae9aa296fd68ed: from storage DS-0eeb57f1-6175-43f3-9611-c11b9699d2f2 node DatanodeRegistration(127.0.0.1:41525, datanodeUuid=c84b3dbf-c391-4f36-b298-3ef2e911d0d9, infoPort=45313, infoSecurePort=0, ipcPort=40173, storageInfo=lv=-57;cid=testClusterID;nsid=994974523;c=1689447921569), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-15 19:05:23,605 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x94a9cb2530a6cb53: Processing first storage report for DS-938bd891-cf5b-4388-85c8-f0af346e90f6 from datanode 1659b223-4fed-4869-94fe-7556636a6e4c 2023-07-15 19:05:23,605 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x94a9cb2530a6cb53: from storage DS-938bd891-cf5b-4388-85c8-f0af346e90f6 node DatanodeRegistration(127.0.0.1:42117, datanodeUuid=1659b223-4fed-4869-94fe-7556636a6e4c, infoPort=36419, infoSecurePort=0, ipcPort=42557, storageInfo=lv=-57;cid=testClusterID;nsid=994974523;c=1689447921569), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 19:05:23,606 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x13ae9aa296fd68ed: Processing first storage report for DS-8357e5d9-5421-45c6-9a75-4594a06728cc from datanode c84b3dbf-c391-4f36-b298-3ef2e911d0d9 2023-07-15 19:05:23,606 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x13ae9aa296fd68ed: from storage DS-8357e5d9-5421-45c6-9a75-4594a06728cc node DatanodeRegistration(127.0.0.1:41525, datanodeUuid=c84b3dbf-c391-4f36-b298-3ef2e911d0d9, infoPort=45313, infoSecurePort=0, ipcPort=40173, storageInfo=lv=-57;cid=testClusterID;nsid=994974523;c=1689447921569), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 19:05:23,606 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x94a9cb2530a6cb53: Processing first storage report for DS-fa233ec9-24e1-406e-af8e-61347268c670 from datanode 1659b223-4fed-4869-94fe-7556636a6e4c 2023-07-15 19:05:23,606 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x94a9cb2530a6cb53: from storage DS-fa233ec9-24e1-406e-af8e-61347268c670 node DatanodeRegistration(127.0.0.1:42117, datanodeUuid=1659b223-4fed-4869-94fe-7556636a6e4c, infoPort=36419, infoSecurePort=0, ipcPort=42557, storageInfo=lv=-57;cid=testClusterID;nsid=994974523;c=1689447921569), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 19:05:23,678 DEBUG [Listener at localhost/42557] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4 2023-07-15 19:05:23,740 INFO [Listener at localhost/42557] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/cluster_a9497dea-7554-f5e1-575e-fef856a8e7af/zookeeper_0, clientPort=54942, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/cluster_a9497dea-7554-f5e1-575e-fef856a8e7af/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/cluster_a9497dea-7554-f5e1-575e-fef856a8e7af/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-15 19:05:23,752 INFO [Listener at localhost/42557] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=54942 2023-07-15 19:05:23,760 INFO [Listener at localhost/42557] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 19:05:23,762 INFO [Listener at localhost/42557] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 19:05:24,429 INFO [Listener at localhost/42557] util.FSUtils(471): Created version file at hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37 with version=8 2023-07-15 19:05:24,429 INFO [Listener at localhost/42557] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/hbase-staging 2023-07-15 19:05:24,745 INFO [Listener at localhost/42557] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-15 19:05:25,305 INFO [Listener at localhost/42557] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 19:05:25,353 INFO [Listener at localhost/42557] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 19:05:25,354 INFO [Listener at localhost/42557] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 19:05:25,355 INFO [Listener at localhost/42557] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 19:05:25,355 INFO [Listener at localhost/42557] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 19:05:25,355 INFO [Listener at localhost/42557] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 19:05:25,503 INFO [Listener at localhost/42557] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 19:05:25,574 DEBUG [Listener at localhost/42557] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-15 19:05:25,669 INFO [Listener at localhost/42557] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38491 2023-07-15 19:05:25,679 INFO [Listener at localhost/42557] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 19:05:25,681 INFO [Listener at localhost/42557] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 19:05:25,701 INFO [Listener at localhost/42557] zookeeper.RecoverableZooKeeper(93): Process identifier=master:38491 connecting to ZooKeeper ensemble=127.0.0.1:54942 2023-07-15 19:05:25,741 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:384910x0, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 19:05:25,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:38491-0x1016a6003100000 connected 2023-07-15 19:05:25,765 DEBUG [Listener at localhost/42557] zookeeper.ZKUtil(164): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 19:05:25,766 DEBUG [Listener at localhost/42557] zookeeper.ZKUtil(164): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 19:05:25,769 DEBUG [Listener at localhost/42557] zookeeper.ZKUtil(164): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 19:05:25,777 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38491 2023-07-15 19:05:25,777 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38491 2023-07-15 19:05:25,777 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38491 2023-07-15 19:05:25,778 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38491 2023-07-15 19:05:25,778 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38491 2023-07-15 19:05:25,784 INFO [Listener at localhost/42557] master.HMaster(444): hbase.rootdir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37, hbase.cluster.distributed=false 2023-07-15 19:05:25,852 INFO [Listener at localhost/42557] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 19:05:25,852 INFO [Listener at localhost/42557] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 19:05:25,852 INFO [Listener at localhost/42557] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 19:05:25,853 INFO [Listener at localhost/42557] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 19:05:25,853 INFO [Listener at localhost/42557] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 19:05:25,853 INFO [Listener at localhost/42557] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 19:05:25,858 INFO [Listener at localhost/42557] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 19:05:25,861 INFO [Listener at localhost/42557] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43223 2023-07-15 19:05:25,863 INFO [Listener at localhost/42557] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 19:05:25,870 DEBUG [Listener at localhost/42557] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 19:05:25,871 INFO [Listener at localhost/42557] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 19:05:25,873 INFO [Listener at localhost/42557] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 19:05:25,875 INFO [Listener at localhost/42557] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43223 connecting to ZooKeeper ensemble=127.0.0.1:54942 2023-07-15 19:05:25,878 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:432230x0, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 19:05:25,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:43223-0x1016a6003100001 connected 2023-07-15 19:05:25,879 DEBUG [Listener at localhost/42557] zookeeper.ZKUtil(164): regionserver:43223-0x1016a6003100001, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 19:05:25,881 DEBUG [Listener at localhost/42557] zookeeper.ZKUtil(164): regionserver:43223-0x1016a6003100001, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 19:05:25,882 DEBUG [Listener at localhost/42557] zookeeper.ZKUtil(164): regionserver:43223-0x1016a6003100001, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 19:05:25,882 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43223 2023-07-15 19:05:25,882 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43223 2023-07-15 19:05:25,883 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43223 2023-07-15 19:05:25,884 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43223 2023-07-15 19:05:25,885 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43223 2023-07-15 19:05:25,898 INFO [Listener at localhost/42557] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 19:05:25,898 INFO [Listener at localhost/42557] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 19:05:25,899 INFO [Listener at localhost/42557] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 19:05:25,899 INFO [Listener at localhost/42557] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 19:05:25,899 INFO [Listener at localhost/42557] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 19:05:25,899 INFO [Listener at localhost/42557] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 19:05:25,899 INFO [Listener at localhost/42557] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 19:05:25,901 INFO [Listener at localhost/42557] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45281 2023-07-15 19:05:25,901 INFO [Listener at localhost/42557] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 19:05:25,903 DEBUG [Listener at localhost/42557] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 19:05:25,903 INFO [Listener at localhost/42557] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 19:05:25,905 INFO [Listener at localhost/42557] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 19:05:25,907 INFO [Listener at localhost/42557] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:45281 connecting to ZooKeeper ensemble=127.0.0.1:54942 2023-07-15 19:05:25,911 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:452810x0, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 19:05:25,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:45281-0x1016a6003100002 connected 2023-07-15 19:05:25,912 DEBUG [Listener at localhost/42557] zookeeper.ZKUtil(164): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 19:05:25,913 DEBUG [Listener at localhost/42557] zookeeper.ZKUtil(164): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 19:05:25,914 DEBUG [Listener at localhost/42557] zookeeper.ZKUtil(164): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 19:05:25,914 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45281 2023-07-15 19:05:25,914 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45281 2023-07-15 19:05:25,915 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45281 2023-07-15 19:05:25,915 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45281 2023-07-15 19:05:25,915 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45281 2023-07-15 19:05:25,917 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,38491,1689447924569 2023-07-15 19:05:25,926 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-15 19:05:25,928 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,38491,1689447924569 2023-07-15 19:05:25,947 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-15 19:05:25,947 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-15 19:05:25,947 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:43223-0x1016a6003100001, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-15 19:05:25,947 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 19:05:25,949 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 19:05:25,950 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,38491,1689447924569 from backup master directory 2023-07-15 19:05:25,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 19:05:25,953 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,38491,1689447924569 2023-07-15 19:05:25,953 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-15 19:05:25,953 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 19:05:25,954 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,38491,1689447924569 2023-07-15 19:05:25,956 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-15 19:05:25,958 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-15 19:05:26,045 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/hbase.id with ID: 3d364618-7816-41f1-b41f-65e5960beece 2023-07-15 19:05:26,090 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 19:05:26,105 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 19:05:26,151 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0c1f39f4 to 127.0.0.1:54942 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 19:05:26,186 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2edbbf40, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 19:05:26,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-15 19:05:26,213 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-15 19:05:26,230 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-15 19:05:26,230 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-15 19:05:26,232 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-15 19:05:26,236 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-15 19:05:26,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 19:05:26,272 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/MasterData/data/master/store-tmp 2023-07-15 19:05:26,313 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 19:05:26,313 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-15 19:05:26,313 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 19:05:26,313 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 19:05:26,313 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-15 19:05:26,313 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 19:05:26,313 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 19:05:26,313 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-15 19:05:26,315 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/MasterData/WALs/jenkins-hbase4.apache.org,38491,1689447924569 2023-07-15 19:05:26,338 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38491%2C1689447924569, suffix=, logDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/MasterData/WALs/jenkins-hbase4.apache.org,38491,1689447924569, archiveDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/MasterData/oldWALs, maxLogs=10 2023-07-15 19:05:26,399 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42117,DS-938bd891-cf5b-4388-85c8-f0af346e90f6,DISK] 2023-07-15 19:05:26,399 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41525,DS-0eeb57f1-6175-43f3-9611-c11b9699d2f2,DISK] 2023-07-15 19:05:26,408 DEBUG [RS-EventLoopGroup-4-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 19:05:26,471 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/MasterData/WALs/jenkins-hbase4.apache.org,38491,1689447924569/jenkins-hbase4.apache.org%2C38491%2C1689447924569.1689447926350 2023-07-15 19:05:26,471 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41525,DS-0eeb57f1-6175-43f3-9611-c11b9699d2f2,DISK], DatanodeInfoWithStorage[127.0.0.1:42117,DS-938bd891-cf5b-4388-85c8-f0af346e90f6,DISK]] 2023-07-15 19:05:26,472 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-15 19:05:26,472 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 19:05:26,475 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-15 19:05:26,476 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-15 19:05:26,532 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-15 19:05:26,540 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-15 19:05:26,565 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-15 19:05:26,577 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:26,583 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-15 19:05:26,584 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-15 19:05:26,599 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-15 19:05:26,603 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 19:05:26,604 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72451321, jitterRate=0.07960881292819977}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 19:05:26,604 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-15 19:05:26,605 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-15 19:05:26,626 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-15 19:05:26,626 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-15 19:05:26,628 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-15 19:05:26,630 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-15 19:05:26,664 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 34 msec 2023-07-15 19:05:26,664 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-15 19:05:26,693 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-15 19:05:26,699 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-15 19:05:26,726 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-15 19:05:26,730 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-15 19:05:26,731 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-15 19:05:26,736 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-15 19:05:26,740 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-15 19:05:26,742 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 19:05:26,744 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-15 19:05:26,744 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-15 19:05:26,756 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-15 19:05:26,760 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-15 19:05:26,760 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:43223-0x1016a6003100001, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-15 19:05:26,760 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-15 19:05:26,761 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 19:05:26,761 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,38491,1689447924569, sessionid=0x1016a6003100000, setting cluster-up flag (Was=false) 2023-07-15 19:05:26,777 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 19:05:26,782 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-15 19:05:26,784 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,38491,1689447924569 2023-07-15 19:05:26,790 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 19:05:26,796 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-15 19:05:26,798 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,38491,1689447924569 2023-07-15 19:05:26,800 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/.hbase-snapshot/.tmp 2023-07-15 19:05:26,819 INFO [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(951): ClusterId : 3d364618-7816-41f1-b41f-65e5960beece 2023-07-15 19:05:26,819 INFO [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(951): ClusterId : 3d364618-7816-41f1-b41f-65e5960beece 2023-07-15 19:05:26,824 DEBUG [RS:1;jenkins-hbase4:45281] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 19:05:26,824 DEBUG [RS:0;jenkins-hbase4:43223] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 19:05:26,830 DEBUG [RS:1;jenkins-hbase4:45281] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 19:05:26,830 DEBUG [RS:0;jenkins-hbase4:43223] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 19:05:26,830 DEBUG [RS:1;jenkins-hbase4:45281] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 19:05:26,830 DEBUG [RS:0;jenkins-hbase4:43223] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 19:05:26,835 DEBUG [RS:0;jenkins-hbase4:43223] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 19:05:26,835 DEBUG [RS:1;jenkins-hbase4:45281] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 19:05:26,836 DEBUG [RS:0;jenkins-hbase4:43223] zookeeper.ReadOnlyZKClient(139): Connect 0x140e3e31 to 127.0.0.1:54942 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 19:05:26,837 DEBUG [RS:1;jenkins-hbase4:45281] zookeeper.ReadOnlyZKClient(139): Connect 0x01bd7640 to 127.0.0.1:54942 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 19:05:26,843 DEBUG [RS:0;jenkins-hbase4:43223] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ac9c2c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 19:05:26,843 DEBUG [RS:1;jenkins-hbase4:45281] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f7a9807, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 19:05:26,843 DEBUG [RS:0;jenkins-hbase4:43223] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ef65541, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 19:05:26,843 DEBUG [RS:1;jenkins-hbase4:45281] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b451d30, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 19:05:26,867 DEBUG [RS:1;jenkins-hbase4:45281] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:45281 2023-07-15 19:05:26,867 DEBUG [RS:0;jenkins-hbase4:43223] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:43223 2023-07-15 19:05:26,871 INFO [RS:0;jenkins-hbase4:43223] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 19:05:26,871 INFO [RS:1;jenkins-hbase4:45281] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 19:05:26,872 INFO [RS:1;jenkins-hbase4:45281] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 19:05:26,871 INFO [RS:0;jenkins-hbase4:43223] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 19:05:26,872 DEBUG [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 19:05:26,872 DEBUG [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 19:05:26,874 INFO [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,38491,1689447924569 with isa=jenkins-hbase4.apache.org/172.31.14.131:43223, startcode=1689447925851 2023-07-15 19:05:26,874 INFO [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,38491,1689447924569 with isa=jenkins-hbase4.apache.org/172.31.14.131:45281, startcode=1689447925898 2023-07-15 19:05:26,894 DEBUG [RS:0;jenkins-hbase4:43223] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 19:05:26,894 DEBUG [RS:1;jenkins-hbase4:45281] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 19:05:26,941 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-15 19:05:26,951 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35643, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 19:05:26,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 19:05:26,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 19:05:26,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 19:05:26,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 19:05:26,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-15 19:05:26,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:26,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 19:05:26,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:26,951 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52569, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 19:05:26,960 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689447956960 2023-07-15 19:05:26,963 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-15 19:05:26,964 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38491] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:579) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 19:05:26,965 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-15 19:05:26,966 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-15 19:05:26,972 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-15 19:05:26,974 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38491] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:579) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 19:05:26,977 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-15 19:05:26,987 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-15 19:05:26,988 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-15 19:05:26,988 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-15 19:05:26,988 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-15 19:05:26,989 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:26,991 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-15 19:05:26,992 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-15 19:05:26,992 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-15 19:05:26,994 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-15 19:05:26,995 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-15 19:05:26,996 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689447926996,5,FailOnTimeoutGroup] 2023-07-15 19:05:26,998 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689447926997,5,FailOnTimeoutGroup] 2023-07-15 19:05:26,998 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:26,998 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-15 19:05:27,000 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,000 DEBUG [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(2830): Master is not running yet 2023-07-15 19:05:27,000 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,000 DEBUG [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(2830): Master is not running yet 2023-07-15 19:05:27,001 WARN [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(1030): reportForDuty failed; sleeping 100 ms and then retrying. 2023-07-15 19:05:27,001 WARN [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(1030): reportForDuty failed; sleeping 100 ms and then retrying. 2023-07-15 19:05:27,020 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-15 19:05:27,021 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-15 19:05:27,021 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37 2023-07-15 19:05:27,038 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 19:05:27,040 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 19:05:27,043 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/info 2023-07-15 19:05:27,044 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 19:05:27,045 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:27,045 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 19:05:27,047 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/rep_barrier 2023-07-15 19:05:27,048 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 19:05:27,049 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:27,049 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 19:05:27,051 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/table 2023-07-15 19:05:27,052 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 19:05:27,053 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:27,054 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740 2023-07-15 19:05:27,055 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740 2023-07-15 19:05:27,059 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 19:05:27,061 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 19:05:27,065 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 19:05:27,066 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72802200, jitterRate=0.08483731746673584}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 19:05:27,066 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 19:05:27,067 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 19:05:27,067 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 19:05:27,067 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 19:05:27,067 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 19:05:27,067 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 19:05:27,068 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 19:05:27,068 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 19:05:27,074 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-15 19:05:27,074 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-15 19:05:27,083 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-15 19:05:27,096 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-15 19:05:27,098 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-15 19:05:27,102 INFO [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,38491,1689447924569 with isa=jenkins-hbase4.apache.org/172.31.14.131:43223, startcode=1689447925851 2023-07-15 19:05:27,102 INFO [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,38491,1689447924569 with isa=jenkins-hbase4.apache.org/172.31.14.131:45281, startcode=1689447925898 2023-07-15 19:05:27,108 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38491] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:27,113 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38491] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,43223,1689447925851 2023-07-15 19:05:27,114 DEBUG [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37 2023-07-15 19:05:27,114 DEBUG [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:45815 2023-07-15 19:05:27,114 DEBUG [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-15 19:05:27,115 DEBUG [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37 2023-07-15 19:05:27,115 DEBUG [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:45815 2023-07-15 19:05:27,115 DEBUG [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-15 19:05:27,124 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 19:05:27,124 DEBUG [RS:0;jenkins-hbase4:43223] zookeeper.ZKUtil(162): regionserver:43223-0x1016a6003100001, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43223,1689447925851 2023-07-15 19:05:27,124 DEBUG [RS:1;jenkins-hbase4:45281] zookeeper.ZKUtil(162): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:27,124 WARN [RS:0;jenkins-hbase4:43223] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 19:05:27,126 WARN [RS:1;jenkins-hbase4:45281] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 19:05:27,126 INFO [RS:0;jenkins-hbase4:43223] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 19:05:27,126 INFO [RS:1;jenkins-hbase4:45281] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 19:05:27,127 DEBUG [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(1948): logDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851 2023-07-15 19:05:27,127 DEBUG [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(1948): logDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:27,127 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,43223,1689447925851] 2023-07-15 19:05:27,128 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,45281,1689447925898] 2023-07-15 19:05:27,137 DEBUG [RS:0;jenkins-hbase4:43223] zookeeper.ZKUtil(162): regionserver:43223-0x1016a6003100001, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43223,1689447925851 2023-07-15 19:05:27,137 DEBUG [RS:1;jenkins-hbase4:45281] zookeeper.ZKUtil(162): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43223,1689447925851 2023-07-15 19:05:27,137 DEBUG [RS:1;jenkins-hbase4:45281] zookeeper.ZKUtil(162): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:27,138 DEBUG [RS:0;jenkins-hbase4:43223] zookeeper.ZKUtil(162): regionserver:43223-0x1016a6003100001, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:27,148 DEBUG [RS:0;jenkins-hbase4:43223] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 19:05:27,148 DEBUG [RS:1;jenkins-hbase4:45281] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 19:05:27,158 INFO [RS:0;jenkins-hbase4:43223] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 19:05:27,158 INFO [RS:1;jenkins-hbase4:45281] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 19:05:27,181 INFO [RS:1;jenkins-hbase4:45281] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 19:05:27,181 INFO [RS:0;jenkins-hbase4:43223] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 19:05:27,186 INFO [RS:0;jenkins-hbase4:43223] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 19:05:27,186 INFO [RS:1;jenkins-hbase4:45281] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 19:05:27,187 INFO [RS:0;jenkins-hbase4:43223] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,187 INFO [RS:1;jenkins-hbase4:45281] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,189 INFO [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 19:05:27,189 INFO [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 19:05:27,198 INFO [RS:0;jenkins-hbase4:43223] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,198 INFO [RS:1;jenkins-hbase4:45281] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,198 DEBUG [RS:1;jenkins-hbase4:45281] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,198 DEBUG [RS:0;jenkins-hbase4:43223] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,199 DEBUG [RS:1;jenkins-hbase4:45281] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,199 DEBUG [RS:0;jenkins-hbase4:43223] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,199 DEBUG [RS:1;jenkins-hbase4:45281] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,199 DEBUG [RS:0;jenkins-hbase4:43223] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,199 DEBUG [RS:1;jenkins-hbase4:45281] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,199 DEBUG [RS:0;jenkins-hbase4:43223] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,199 DEBUG [RS:1;jenkins-hbase4:45281] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,199 DEBUG [RS:0;jenkins-hbase4:43223] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,199 DEBUG [RS:1;jenkins-hbase4:45281] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 19:05:27,199 DEBUG [RS:0;jenkins-hbase4:43223] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 19:05:27,199 DEBUG [RS:1;jenkins-hbase4:45281] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,199 DEBUG [RS:0;jenkins-hbase4:43223] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,200 DEBUG [RS:1;jenkins-hbase4:45281] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,200 DEBUG [RS:0;jenkins-hbase4:43223] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,200 DEBUG [RS:1;jenkins-hbase4:45281] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,200 DEBUG [RS:0;jenkins-hbase4:43223] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,200 DEBUG [RS:1;jenkins-hbase4:45281] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,200 DEBUG [RS:0;jenkins-hbase4:43223] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:27,210 INFO [RS:1;jenkins-hbase4:45281] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,211 INFO [RS:1;jenkins-hbase4:45281] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,211 INFO [RS:1;jenkins-hbase4:45281] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,215 INFO [RS:0;jenkins-hbase4:43223] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,216 INFO [RS:0;jenkins-hbase4:43223] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,216 INFO [RS:0;jenkins-hbase4:43223] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,233 INFO [RS:0;jenkins-hbase4:43223] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 19:05:27,233 INFO [RS:1;jenkins-hbase4:45281] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 19:05:27,236 INFO [RS:0;jenkins-hbase4:43223] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43223,1689447925851-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,236 INFO [RS:1;jenkins-hbase4:45281] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45281,1689447925898-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,250 DEBUG [jenkins-hbase4:38491] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-15 19:05:27,255 INFO [RS:0;jenkins-hbase4:43223] regionserver.Replication(203): jenkins-hbase4.apache.org,43223,1689447925851 started 2023-07-15 19:05:27,255 INFO [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,43223,1689447925851, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:43223, sessionid=0x1016a6003100001 2023-07-15 19:05:27,255 DEBUG [RS:0;jenkins-hbase4:43223] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 19:05:27,255 DEBUG [RS:0;jenkins-hbase4:43223] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,43223,1689447925851 2023-07-15 19:05:27,255 DEBUG [RS:0;jenkins-hbase4:43223] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43223,1689447925851' 2023-07-15 19:05:27,255 DEBUG [RS:0;jenkins-hbase4:43223] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 19:05:27,255 DEBUG [jenkins-hbase4:38491] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 19:05:27,256 DEBUG [RS:0;jenkins-hbase4:43223] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 19:05:27,257 DEBUG [RS:0;jenkins-hbase4:43223] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 19:05:27,257 DEBUG [RS:0;jenkins-hbase4:43223] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 19:05:27,257 DEBUG [RS:0;jenkins-hbase4:43223] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,43223,1689447925851 2023-07-15 19:05:27,257 DEBUG [RS:0;jenkins-hbase4:43223] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43223,1689447925851' 2023-07-15 19:05:27,257 DEBUG [RS:0;jenkins-hbase4:43223] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 19:05:27,257 DEBUG [RS:0;jenkins-hbase4:43223] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 19:05:27,258 INFO [RS:1;jenkins-hbase4:45281] regionserver.Replication(203): jenkins-hbase4.apache.org,45281,1689447925898 started 2023-07-15 19:05:27,258 INFO [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,45281,1689447925898, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:45281, sessionid=0x1016a6003100002 2023-07-15 19:05:27,258 DEBUG [RS:1;jenkins-hbase4:45281] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 19:05:27,258 DEBUG [RS:0;jenkins-hbase4:43223] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 19:05:27,258 DEBUG [RS:1;jenkins-hbase4:45281] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:27,259 INFO [RS:0;jenkins-hbase4:43223] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 19:05:27,259 DEBUG [RS:1;jenkins-hbase4:45281] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45281,1689447925898' 2023-07-15 19:05:27,260 DEBUG [RS:1;jenkins-hbase4:45281] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 19:05:27,259 INFO [RS:0;jenkins-hbase4:43223] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 19:05:27,260 DEBUG [RS:1;jenkins-hbase4:45281] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 19:05:27,261 DEBUG [RS:1;jenkins-hbase4:45281] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 19:05:27,261 DEBUG [RS:1;jenkins-hbase4:45281] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 19:05:27,261 DEBUG [RS:1;jenkins-hbase4:45281] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:27,261 DEBUG [RS:1;jenkins-hbase4:45281] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45281,1689447925898' 2023-07-15 19:05:27,261 DEBUG [RS:1;jenkins-hbase4:45281] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 19:05:27,263 DEBUG [RS:1;jenkins-hbase4:45281] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 19:05:27,263 DEBUG [RS:1;jenkins-hbase4:45281] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 19:05:27,263 INFO [RS:1;jenkins-hbase4:45281] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 19:05:27,264 INFO [RS:1;jenkins-hbase4:45281] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 19:05:27,265 DEBUG [jenkins-hbase4:38491] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 19:05:27,265 DEBUG [jenkins-hbase4:38491] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 19:05:27,265 DEBUG [jenkins-hbase4:38491] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 19:05:27,268 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43223,1689447925851, state=OPENING 2023-07-15 19:05:27,275 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-15 19:05:27,276 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 19:05:27,277 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 19:05:27,282 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43223,1689447925851}] 2023-07-15 19:05:27,371 INFO [RS:0;jenkins-hbase4:43223] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43223%2C1689447925851, suffix=, logDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851, archiveDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/oldWALs, maxLogs=32 2023-07-15 19:05:27,371 INFO [RS:1;jenkins-hbase4:45281] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45281%2C1689447925898, suffix=, logDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898, archiveDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/oldWALs, maxLogs=32 2023-07-15 19:05:27,395 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42117,DS-938bd891-cf5b-4388-85c8-f0af346e90f6,DISK] 2023-07-15 19:05:27,395 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41525,DS-0eeb57f1-6175-43f3-9611-c11b9699d2f2,DISK] 2023-07-15 19:05:27,396 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41525,DS-0eeb57f1-6175-43f3-9611-c11b9699d2f2,DISK] 2023-07-15 19:05:27,396 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42117,DS-938bd891-cf5b-4388-85c8-f0af346e90f6,DISK] 2023-07-15 19:05:27,405 INFO [RS:0;jenkins-hbase4:43223] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851/jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375 2023-07-15 19:05:27,405 INFO [RS:1;jenkins-hbase4:45281] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898/jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375 2023-07-15 19:05:27,405 DEBUG [RS:0;jenkins-hbase4:43223] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42117,DS-938bd891-cf5b-4388-85c8-f0af346e90f6,DISK], DatanodeInfoWithStorage[127.0.0.1:41525,DS-0eeb57f1-6175-43f3-9611-c11b9699d2f2,DISK]] 2023-07-15 19:05:27,405 DEBUG [RS:1;jenkins-hbase4:45281] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41525,DS-0eeb57f1-6175-43f3-9611-c11b9699d2f2,DISK], DatanodeInfoWithStorage[127.0.0.1:42117,DS-938bd891-cf5b-4388-85c8-f0af346e90f6,DISK]] 2023-07-15 19:05:27,468 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,43223,1689447925851 2023-07-15 19:05:27,470 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 19:05:27,474 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 19:05:27,485 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-15 19:05:27,485 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 19:05:27,488 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta, suffix=.meta, logDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851, archiveDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/oldWALs, maxLogs=32 2023-07-15 19:05:27,506 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42117,DS-938bd891-cf5b-4388-85c8-f0af346e90f6,DISK] 2023-07-15 19:05:27,507 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41525,DS-0eeb57f1-6175-43f3-9611-c11b9699d2f2,DISK] 2023-07-15 19:05:27,514 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851/jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta 2023-07-15 19:05:27,516 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42117,DS-938bd891-cf5b-4388-85c8-f0af346e90f6,DISK], DatanodeInfoWithStorage[127.0.0.1:41525,DS-0eeb57f1-6175-43f3-9611-c11b9699d2f2,DISK]] 2023-07-15 19:05:27,516 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-15 19:05:27,519 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 19:05:27,541 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-15 19:05:27,570 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-15 19:05:27,575 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-15 19:05:27,576 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 19:05:27,576 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-15 19:05:27,576 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-15 19:05:27,579 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 19:05:27,581 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/info 2023-07-15 19:05:27,581 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/info 2023-07-15 19:05:27,581 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 19:05:27,582 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:27,583 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 19:05:27,584 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/rep_barrier 2023-07-15 19:05:27,584 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/rep_barrier 2023-07-15 19:05:27,584 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 19:05:27,585 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:27,585 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 19:05:27,587 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/table 2023-07-15 19:05:27,587 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/table 2023-07-15 19:05:27,588 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 19:05:27,588 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:27,590 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740 2023-07-15 19:05:27,593 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740 2023-07-15 19:05:27,596 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 19:05:27,598 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 19:05:27,600 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67259891, jitterRate=0.00225047767162323}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 19:05:27,600 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 19:05:27,609 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689447927460 2023-07-15 19:05:27,628 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-15 19:05:27,629 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-15 19:05:27,629 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43223,1689447925851, state=OPEN 2023-07-15 19:05:27,633 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 19:05:27,633 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 19:05:27,637 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-15 19:05:27,638 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43223,1689447925851 in 351 msec 2023-07-15 19:05:27,642 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-15 19:05:27,642 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 555 msec 2023-07-15 19:05:27,649 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 793 msec 2023-07-15 19:05:27,649 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689447927649, completionTime=-1 2023-07-15 19:05:27,649 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-15 19:05:27,650 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-15 19:05:27,713 DEBUG [hconnection-0x19362f23-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 19:05:27,716 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42328, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 19:05:27,734 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-15 19:05:27,734 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689447987734 2023-07-15 19:05:27,734 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689448047734 2023-07-15 19:05:27,734 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 84 msec 2023-07-15 19:05:27,760 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38491,1689447924569-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,760 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38491,1689447924569-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,760 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38491,1689447924569-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,762 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:38491, period=300000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,762 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:27,768 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-15 19:05:27,777 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-15 19:05:27,778 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-15 19:05:27,788 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-15 19:05:27,790 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-15 19:05:27,793 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-15 19:05:27,816 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/.tmp/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:27,818 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/.tmp/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0 empty. 2023-07-15 19:05:27,819 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/.tmp/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:27,819 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-15 19:05:27,846 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-15 19:05:27,849 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 24f2c4350af36d0c0546d743c16c75c0, NAME => 'hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/.tmp 2023-07-15 19:05:27,868 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 19:05:27,868 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 24f2c4350af36d0c0546d743c16c75c0, disabling compactions & flushes 2023-07-15 19:05:27,868 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:27,868 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:27,868 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. after waiting 0 ms 2023-07-15 19:05:27,868 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:27,868 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:27,868 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 24f2c4350af36d0c0546d743c16c75c0: 2023-07-15 19:05:27,873 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-15 19:05:27,888 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689447927875"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689447927875"}]},"ts":"1689447927875"} 2023-07-15 19:05:27,916 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-15 19:05:27,918 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-15 19:05:27,923 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689447927919"}]},"ts":"1689447927919"} 2023-07-15 19:05:27,929 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-15 19:05:27,933 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 19:05:27,934 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 19:05:27,934 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 19:05:27,934 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 19:05:27,937 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=24f2c4350af36d0c0546d743c16c75c0, ASSIGN}] 2023-07-15 19:05:27,939 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=24f2c4350af36d0c0546d743c16c75c0, ASSIGN 2023-07-15 19:05:27,942 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=24f2c4350af36d0c0546d743c16c75c0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45281,1689447925898; forceNewPlan=false, retain=false 2023-07-15 19:05:28,094 INFO [jenkins-hbase4:38491] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-15 19:05:28,095 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=24f2c4350af36d0c0546d743c16c75c0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:28,095 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689447928095"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689447928095"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689447928095"}]},"ts":"1689447928095"} 2023-07-15 19:05:28,101 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 24f2c4350af36d0c0546d743c16c75c0, server=jenkins-hbase4.apache.org,45281,1689447925898}] 2023-07-15 19:05:28,254 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:28,255 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 19:05:28,257 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43140, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 19:05:28,264 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:28,264 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 24f2c4350af36d0c0546d743c16c75c0, NAME => 'hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0.', STARTKEY => '', ENDKEY => ''} 2023-07-15 19:05:28,265 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:28,265 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 19:05:28,265 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:28,265 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:28,268 INFO [StoreOpener-24f2c4350af36d0c0546d743c16c75c0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:28,271 DEBUG [StoreOpener-24f2c4350af36d0c0546d743c16c75c0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/info 2023-07-15 19:05:28,271 DEBUG [StoreOpener-24f2c4350af36d0c0546d743c16c75c0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/info 2023-07-15 19:05:28,271 INFO [StoreOpener-24f2c4350af36d0c0546d743c16c75c0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 24f2c4350af36d0c0546d743c16c75c0 columnFamilyName info 2023-07-15 19:05:28,272 INFO [StoreOpener-24f2c4350af36d0c0546d743c16c75c0-1] regionserver.HStore(310): Store=24f2c4350af36d0c0546d743c16c75c0/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:28,273 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:28,274 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:28,279 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:28,282 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 19:05:28,283 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 24f2c4350af36d0c0546d743c16c75c0; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68351744, jitterRate=0.018520355224609375}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 19:05:28,283 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 24f2c4350af36d0c0546d743c16c75c0: 2023-07-15 19:05:28,285 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0., pid=6, masterSystemTime=1689447928254 2023-07-15 19:05:28,290 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:28,290 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:28,292 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=24f2c4350af36d0c0546d743c16c75c0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:28,292 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689447928291"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689447928291"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689447928291"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689447928291"}]},"ts":"1689447928291"} 2023-07-15 19:05:28,299 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-15 19:05:28,299 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 24f2c4350af36d0c0546d743c16c75c0, server=jenkins-hbase4.apache.org,45281,1689447925898 in 195 msec 2023-07-15 19:05:28,303 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-15 19:05:28,304 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=24f2c4350af36d0c0546d743c16c75c0, ASSIGN in 362 msec 2023-07-15 19:05:28,305 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-15 19:05:28,305 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689447928305"}]},"ts":"1689447928305"} 2023-07-15 19:05:28,308 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-15 19:05:28,312 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-15 19:05:28,315 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 534 msec 2023-07-15 19:05:28,392 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-15 19:05:28,393 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-15 19:05:28,393 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 19:05:28,414 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 19:05:28,416 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43144, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 19:05:28,430 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-15 19:05:28,449 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-15 19:05:28,455 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 31 msec 2023-07-15 19:05:28,463 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-15 19:05:28,476 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-15 19:05:28,481 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-07-15 19:05:28,489 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-15 19:05:28,492 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-15 19:05:28,494 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.540sec 2023-07-15 19:05:28,497 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-15 19:05:28,498 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-15 19:05:28,498 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-15 19:05:28,499 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38491,1689447924569-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-15 19:05:28,500 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38491,1689447924569-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-15 19:05:28,509 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-15 19:05:28,524 DEBUG [Listener at localhost/42557] zookeeper.ReadOnlyZKClient(139): Connect 0x04ca72c7 to 127.0.0.1:54942 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 19:05:28,529 DEBUG [Listener at localhost/42557] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ca962b9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 19:05:28,542 DEBUG [hconnection-0x31531590-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 19:05:28,553 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42334, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 19:05:28,564 INFO [Listener at localhost/42557] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,38491,1689447924569 2023-07-15 19:05:28,573 DEBUG [Listener at localhost/42557] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-15 19:05:28,576 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48594, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-15 19:05:28,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38491] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-15 19:05:28,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38491] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-15 19:05:28,593 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-15 19:05:28,594 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-15 19:05:28,597 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/.tmp/data/default/t/44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:28,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38491] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-15 19:05:28,598 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/.tmp/data/default/t/44c8b933b39ea824f34757f7c29e20d7 empty. 2023-07-15 19:05:28,600 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/.tmp/data/default/t/44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:28,600 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-15 19:05:28,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38491] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-15 19:05:28,619 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-15 19:05:28,621 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => 44c8b933b39ea824f34757f7c29e20d7, NAME => 't,,1689447928585.44c8b933b39ea824f34757f7c29e20d7.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/.tmp 2023-07-15 19:05:28,637 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 19:05:28,637 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing 44c8b933b39ea824f34757f7c29e20d7, disabling compactions & flushes 2023-07-15 19:05:28,637 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:28,637 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:28,637 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. after waiting 0 ms 2023-07-15 19:05:28,637 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:28,637 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:28,637 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for 44c8b933b39ea824f34757f7c29e20d7: 2023-07-15 19:05:28,641 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-15 19:05:28,643 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689447928642"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689447928642"}]},"ts":"1689447928642"} 2023-07-15 19:05:28,645 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-15 19:05:28,647 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-15 19:05:28,647 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689447928647"}]},"ts":"1689447928647"} 2023-07-15 19:05:28,649 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-15 19:05:28,659 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 19:05:28,660 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 19:05:28,660 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 19:05:28,660 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 19:05:28,660 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=44c8b933b39ea824f34757f7c29e20d7, ASSIGN}] 2023-07-15 19:05:28,662 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=44c8b933b39ea824f34757f7c29e20d7, ASSIGN 2023-07-15 19:05:28,664 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=44c8b933b39ea824f34757f7c29e20d7, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43223,1689447925851; forceNewPlan=false, retain=false 2023-07-15 19:05:28,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38491] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-15 19:05:28,814 INFO [jenkins-hbase4:38491] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-15 19:05:28,815 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=44c8b933b39ea824f34757f7c29e20d7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43223,1689447925851 2023-07-15 19:05:28,816 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689447928815"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689447928815"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689447928815"}]},"ts":"1689447928815"} 2023-07-15 19:05:28,819 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 44c8b933b39ea824f34757f7c29e20d7, server=jenkins-hbase4.apache.org,43223,1689447925851}] 2023-07-15 19:05:28,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38491] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-15 19:05:28,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:28,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 44c8b933b39ea824f34757f7c29e20d7, NAME => 't,,1689447928585.44c8b933b39ea824f34757f7c29e20d7.', STARTKEY => '', ENDKEY => ''} 2023-07-15 19:05:28,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:28,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 19:05:28,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:28,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:28,981 INFO [StoreOpener-44c8b933b39ea824f34757f7c29e20d7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:28,983 DEBUG [StoreOpener-44c8b933b39ea824f34757f7c29e20d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/f 2023-07-15 19:05:28,983 DEBUG [StoreOpener-44c8b933b39ea824f34757f7c29e20d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/f 2023-07-15 19:05:28,984 INFO [StoreOpener-44c8b933b39ea824f34757f7c29e20d7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44c8b933b39ea824f34757f7c29e20d7 columnFamilyName f 2023-07-15 19:05:28,985 INFO [StoreOpener-44c8b933b39ea824f34757f7c29e20d7-1] regionserver.HStore(310): Store=44c8b933b39ea824f34757f7c29e20d7/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:28,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:28,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:28,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:28,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 19:05:28,995 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 44c8b933b39ea824f34757f7c29e20d7; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59653250, jitterRate=-0.1110973060131073}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 19:05:28,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 44c8b933b39ea824f34757f7c29e20d7: 2023-07-15 19:05:28,996 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7., pid=11, masterSystemTime=1689447928972 2023-07-15 19:05:28,999 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:28,999 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:29,000 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=44c8b933b39ea824f34757f7c29e20d7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43223,1689447925851 2023-07-15 19:05:29,000 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689447928999"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689447928999"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689447928999"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689447928999"}]},"ts":"1689447928999"} 2023-07-15 19:05:29,006 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-15 19:05:29,006 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 44c8b933b39ea824f34757f7c29e20d7, server=jenkins-hbase4.apache.org,43223,1689447925851 in 184 msec 2023-07-15 19:05:29,009 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-15 19:05:29,009 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=44c8b933b39ea824f34757f7c29e20d7, ASSIGN in 346 msec 2023-07-15 19:05:29,010 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-15 19:05:29,010 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689447929010"}]},"ts":"1689447929010"} 2023-07-15 19:05:29,012 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-15 19:05:29,015 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-15 19:05:29,017 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 426 msec 2023-07-15 19:05:29,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38491] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-15 19:05:29,220 INFO [Listener at localhost/42557] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-15 19:05:29,220 DEBUG [Listener at localhost/42557] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-15 19:05:29,221 INFO [Listener at localhost/42557] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 19:05:29,225 INFO [Listener at localhost/42557] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-15 19:05:29,226 INFO [Listener at localhost/42557] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 19:05:29,226 INFO [Listener at localhost/42557] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-15 19:05:29,272 INFO [Listener at localhost/42557] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=294, OpenFileDescriptor=585, MaxFileDescriptor=60000, SystemLoadAverage=223, ProcessCount=170, AvailableMemoryMB=6212 2023-07-15 19:05:29,274 INFO [Listener at localhost/42557] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-15 19:05:29,277 DEBUG [Listener at localhost/42557] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-15 19:05:29,278 DEBUG [Listener at localhost/42557] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:29,279 INFO [Listener at localhost/42557] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-15 19:05:29,279 INFO [Listener at localhost/42557] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-15 19:05:29,282 INFO [Listener at localhost/42557] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689447928305/Put/vlen=2/seqid=0} 2023-07-15 19:05:29,282 INFO [Listener at localhost/42557] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-15 19:05:29,282 INFO [Listener at localhost/42557] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0./info:regioninfo/1689447928291/Put/vlen=41/seqid=0, hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0./info:seqnumDuringOpen/1689447928291/Put/vlen=8/seqid=0, hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0./info:server/1689447928291/Put/vlen=31/seqid=0, hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0./info:serverstartcode/1689447928291/Put/vlen=8/seqid=0, hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0./info:sn/1689447928095/Put/vlen=45/seqid=0, hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0./info:state/1689447928291/Put/vlen=4/seqid=0} 2023-07-15 19:05:29,283 INFO [Listener at localhost/42557] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 24f2c4350af36d0c0546d743c16c75c0, NAME => 'hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0.', STARTKEY => '', ENDKEY => ''} 2023-07-15 19:05:29,283 INFO [Listener at localhost/42557] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689447929010/Put/vlen=2/seqid=0} 2023-07-15 19:05:29,283 INFO [Listener at localhost/42557] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-15 19:05:29,283 INFO [Listener at localhost/42557] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7./info:regioninfo/1689447928999/Put/vlen=35/seqid=0, t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7./info:seqnumDuringOpen/1689447928999/Put/vlen=8/seqid=0, t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7./info:server/1689447928999/Put/vlen=31/seqid=0, t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7./info:serverstartcode/1689447928999/Put/vlen=8/seqid=0, t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7./info:sn/1689447928815/Put/vlen=45/seqid=0, t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7./info:state/1689447928999/Put/vlen=4/seqid=0} 2023-07-15 19:05:29,283 INFO [Listener at localhost/42557] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 44c8b933b39ea824f34757f7c29e20d7, NAME => 't,,1689447928585.44c8b933b39ea824f34757f7c29e20d7.', STARTKEY => '', ENDKEY => ''} 2023-07-15 19:05:29,285 DEBUG [Listener at localhost/42557] zookeeper.ReadOnlyZKClient(139): Connect 0x7c11e80b to 127.0.0.1:54942 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 19:05:29,289 DEBUG [Listener at localhost/42557] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7944646, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 19:05:29,289 INFO [Listener at localhost/42557] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-15 19:05:29,290 INFO [Listener at localhost/42557] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-15 19:05:29,292 DEBUG [hconnection-0x1e8cbdf7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 19:05:29,296 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42338, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 19:05:29,301 INFO [Listener at localhost/42557] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-15 19:05:29,305 ERROR [Listener at localhost/42557] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,43223,1689447925851: die! ***** 2023-07-15 19:05:29,306 ERROR [Listener at localhost/42557] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-15 19:05:29,309 DEBUG [Listener at localhost/42557] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-15 19:05:29,312 DEBUG [Listener at localhost/42557] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-15 19:05:29,312 DEBUG [Listener at localhost/42557] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-15 19:05:29,313 DEBUG [Listener at localhost/42557] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-15 19:05:29,313 INFO [Listener at localhost/42557] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 884473856, "init": 513802240, "max": 2051014656, "used": 390716112 }, "NonHeapMemoryUsage": { "committed": 113205248, "init": 2555904, "max": -1, "used": 111261768 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-15 19:05:29,318 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=38491] master.MasterRpcServices(612): jenkins-hbase4.apache.org,43223,1689447925851 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,43223,1689447925851: die! ***** 2023-07-15 19:05:29,321 INFO [Listener at localhost/42557] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,43223,1689447925851' ***** 2023-07-15 19:05:29,321 INFO [Listener at localhost/42557] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-15 19:05:29,321 INFO [RS:0;jenkins-hbase4:43223] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 19:05:29,322 INFO [RS:0;jenkins-hbase4:43223] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-15 19:05:29,322 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 19:05:29,322 INFO [RS:0;jenkins-hbase4:43223] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-15 19:05:29,322 INFO [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(3305): Received CLOSE for 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:29,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43223] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner -6380134443163582453 2023-07-15 19:05:29,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43223] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43223,1689447925851 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 19:05:29,323 INFO [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,43223,1689447925851 2023-07-15 19:05:29,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43223] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 172.31.14.131:42338 deadline: 1689447989322, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43223,1689447925851 aborting 2023-07-15 19:05:29,324 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 44c8b933b39ea824f34757f7c29e20d7, disabling compactions & flushes 2023-07-15 19:05:29,324 DEBUG [RS:0;jenkins-hbase4:43223] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x140e3e31 to 127.0.0.1:54942 2023-07-15 19:05:29,324 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:29,324 DEBUG [RS:0;jenkins-hbase4:43223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 19:05:29,324 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:29,324 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. after waiting 0 ms 2023-07-15 19:05:29,324 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:29,324 INFO [RS:0;jenkins-hbase4:43223] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-15 19:05:29,324 INFO [RS:0;jenkins-hbase4:43223] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-15 19:05:29,324 INFO [RS:0;jenkins-hbase4:43223] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-15 19:05:29,324 INFO [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-15 19:05:29,325 DEBUG [Listener at localhost/42557] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause 2023-07-15 19:05:29,325 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) 2023-07-15 19:05:29,325 INFO [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(1474): Waiting on 2 regions to close at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-15 19:05:29,326 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43223,1689447925851 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more 2023-07-15 19:05:29,326 DEBUG [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(1478): Online Regions={44c8b933b39ea824f34757f7c29e20d7=t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7., 1588230740=hbase:meta,,1.1588230740} 2023-07-15 19:05:29,327 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43223,1689447925851 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) 2023-07-15 19:05:29,327 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) 2023-07-15 19:05:29,327 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 44c8b933b39ea824f34757f7c29e20d7: at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) 2023-07-15 19:05:29,327 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) 2023-07-15 19:05:29,327 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) 2023-07-15 19:05:29,327 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 ... 1 more 2023-07-15 19:05:29,327 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:29,328 DEBUG [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-15 19:05:29,329 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 2394 in region hbase:meta,,1.1588230740 2023-07-15 19:05:29,329 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 19:05:29,329 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 19:05:29,329 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 19:05:29,330 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-15 19:05:29,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43223] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43223,1689447925851 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 19:05:29,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43223] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:42338 deadline: 1689447989330, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43223,1689447925851 aborting 2023-07-15 19:05:29,434 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43223] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43223,1689447925851 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2513) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44992) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 19:05:29,434 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43223] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:42338 deadline: 1689447989434, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43223,1689447925851 aborting 2023-07-15 19:05:29,529 INFO [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,43223,1689447925851; all regions closed. 2023-07-15 19:05:29,539 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43223] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43223,1689447925851 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2513) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44992) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 19:05:29,539 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=43223] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:42338 deadline: 1689447989539, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,43223,1689447925851 aborting 2023-07-15 19:05:29,543 DEBUG [RS:0;jenkins-hbase4:43223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 19:05:29,544 INFO [RS:0;jenkins-hbase4:43223] regionserver.LeaseManager(133): Closed leases 2023-07-15 19:05:29,544 INFO [RS:0;jenkins-hbase4:43223] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-15 19:05:29,544 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 19:05:29,545 INFO [RS:0;jenkins-hbase4:43223] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43223 2023-07-15 19:05:29,554 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:43223-0x1016a6003100001, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43223,1689447925851 2023-07-15 19:05:29,554 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 19:05:29,554 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:43223-0x1016a6003100001, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 19:05:29,554 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43223,1689447925851 2023-07-15 19:05:29,555 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 19:05:29,556 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,43223,1689447925851] 2023-07-15 19:05:29,556 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,43223,1689447925851; numProcessing=1 2023-07-15 19:05:29,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:29,557 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,43223,1689447925851 znode expired, triggering replicatorRemoved event 2023-07-15 19:05:29,558 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,43223,1689447925851 already deleted, retry=false 2023-07-15 19:05:29,558 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,43223,1689447925851 on jenkins-hbase4.apache.org,38491,1689447924569 2023-07-15 19:05:29,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:29,566 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,43223,1689447925851, splitWal=true, meta=true 2023-07-15 19:05:29,567 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,43223,1689447925851 (carryingMeta=true) jenkins-hbase4.apache.org,43223,1689447925851/CRASHED/regionCount=2/lock=java.util.concurrent.locks.ReentrantReadWriteLock@2e6dfde[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-15 19:05:29,569 INFO [PEWorker-2] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,43223,1689447925851, splitWal=true, meta=true 2023-07-15 19:05:29,571 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,43223,1689447925851, splitWal=true, meta=true, isMeta: true 2023-07-15 19:05:29,573 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting 2023-07-15 19:05:29,575 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,43223,1689447925851 WAL count=1, meta=true 2023-07-15 19:05:29,579 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta}] 2023-07-15 19:05:29,587 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:29,589 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta, worker=jenkins-hbase4.apache.org,45281,1689447925898}] 2023-07-15 19:05:29,656 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:43223-0x1016a6003100001, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 19:05:29,657 INFO [RS:0;jenkins-hbase4:43223] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,43223,1689447925851; zookeeper connection closed. 2023-07-15 19:05:29,657 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:43223-0x1016a6003100001, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 19:05:29,657 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3114aca0] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3114aca0 2023-07-15 19:05:29,743 WARN [RPCClient-NioEventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:43223 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43223 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 19:05:29,744 DEBUG [RPCClient-NioEventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:43223 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43223 2023-07-15 19:05:29,753 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45281] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-15 19:05:29,775 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta, size=3.1 K (3185bytes) 2023-07-15 19:05:29,776 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta 2023-07-15 19:05:29,811 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta after 35ms 2023-07-15 19:05:29,818 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta took 44ms 2023-07-15 19:05:29,832 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-15 19:05:29,835 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta so closing down 2023-07-15 19:05:29,835 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 19:05:29,839 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-15 19:05:29,841 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta.temp 2023-07-15 19:05:29,843 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 19:05:29,854 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-15 19:05:29,856 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta.temp to hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-15 19:05:29,856 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 37 ms; skipped=1; WAL=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false 2023-07-15 19:05:29,856 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta, journal: Splitting hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta, size=3.1 K (3185bytes) at 1689447929775Finishing writing output for hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta so closing down at 1689447929835 (+60 ms)Creating recovered edits writer path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta.temp at 1689447929842 (+7 ms)3 split writer threads finished at 1689447929843 (+1 ms)Closed recovered edits writer path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689447929854 (+11 ms)Rename recovered edits hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta.temp to hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 at 1689447929856 (+2 ms)Processed 11 edits across 1 Regions in 37 ms; skipped=1; WAL=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false at 1689447929856 2023-07-15 19:05:29,857 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta 2023-07-15 19:05:29,859 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-15 19:05:29,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38491] master.HMaster(3641): Remote procedure done, pid=14 2023-07-15 19:05:29,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38491] wal.WALSplitUtil(141): Moved hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta to hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/oldWALs 2023-07-15 19:05:29,871 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-15 19:05:29,871 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta, worker=jenkins-hbase4.apache.org,45281,1689447925898 in 279 msec 2023-07-15 19:05:29,873 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:29,876 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-15 19:05:29,876 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C43223%2C1689447925851.meta.1689447927490.meta, worker=jenkins-hbase4.apache.org,45281,1689447925898 in 296 msec 2023-07-15 19:05:29,879 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting dir is empty, no logs to split. 2023-07-15 19:05:29,879 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,43223,1689447925851 WAL count=0, meta=true 2023-07-15 19:05:29,879 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,43223,1689447925851 WAL splitting is done? wals=0, meta=true 2023-07-15 19:05:29,880 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-15 19:05:29,881 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-15 19:05:29,882 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 19:05:30,033 DEBUG [jenkins-hbase4:38491] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-15 19:05:30,033 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,45281,1689447925898, state=OPENING 2023-07-15 19:05:30,036 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 19:05:30,036 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 19:05:30,036 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,45281,1689447925898}] 2023-07-15 19:05:30,045 WARN [ReadOnlyZKClient-127.0.0.1:54942@0x7c11e80b] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-15 19:05:30,046 DEBUG [hconnection-0x1e8cbdf7-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 19:05:30,048 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43146, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 19:05:30,049 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=45281] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:43146 deadline: 1689447990049, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:30,195 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-15 19:05:30,195 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 19:05:30,197 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta, suffix=.meta, logDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898, archiveDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/oldWALs, maxLogs=32 2023-07-15 19:05:30,216 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42117,DS-938bd891-cf5b-4388-85c8-f0af346e90f6,DISK] 2023-07-15 19:05:30,217 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 19:05:30,218 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41525,DS-0eeb57f1-6175-43f3-9611-c11b9699d2f2,DISK] 2023-07-15 19:05:30,221 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898/jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta 2023-07-15 19:05:30,221 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42117,DS-938bd891-cf5b-4388-85c8-f0af346e90f6,DISK], DatanodeInfoWithStorage[127.0.0.1:41525,DS-0eeb57f1-6175-43f3-9611-c11b9699d2f2,DISK]] 2023-07-15 19:05:30,221 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-15 19:05:30,221 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 19:05:30,221 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-15 19:05:30,222 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-15 19:05:30,222 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-15 19:05:30,222 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 19:05:30,222 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-15 19:05:30,222 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-15 19:05:30,224 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 19:05:30,225 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/info 2023-07-15 19:05:30,225 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/info 2023-07-15 19:05:30,226 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 19:05:30,227 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:30,227 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 19:05:30,228 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/rep_barrier 2023-07-15 19:05:30,228 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/rep_barrier 2023-07-15 19:05:30,229 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 19:05:30,229 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:30,229 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 19:05:30,230 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/table 2023-07-15 19:05:30,231 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/table 2023-07-15 19:05:30,231 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 19:05:30,232 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:30,233 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740 2023-07-15 19:05:30,236 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740 2023-07-15 19:05:30,238 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-15 19:05:30,246 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 24, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-15 19:05:30,246 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=2.34 KB heapSize=4.86 KB 2023-07-15 19:05:30,298 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.18 KB at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/.tmp/info/54303d9474584fa398c46f9c8226c151 2023-07-15 19:05:30,350 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/.tmp/table/4c97bbfc44c247bda10817d8b84d47e0 2023-07-15 19:05:30,358 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/.tmp/info/54303d9474584fa398c46f9c8226c151 as hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/info/54303d9474584fa398c46f9c8226c151 2023-07-15 19:05:30,366 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/info/54303d9474584fa398c46f9c8226c151, entries=20, sequenceid=13, filesize=6.9 K 2023-07-15 19:05:30,368 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/.tmp/table/4c97bbfc44c247bda10817d8b84d47e0 as hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/table/4c97bbfc44c247bda10817d8b84d47e0 2023-07-15 19:05:30,376 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/table/4c97bbfc44c247bda10817d8b84d47e0, entries=4, sequenceid=13, filesize=4.7 K 2023-07-15 19:05:30,376 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.34 KB/2394, heapSize ~4.58 KB/4688, currentSize=0 B/0 for 1588230740 in 130ms, sequenceid=13, compaction requested=false; wal=null 2023-07-15 19:05:30,378 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-15 19:05:30,381 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-15 19:05:30,383 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 19:05:30,385 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 19:05:30,388 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-15 19:05:30,389 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72994638, jitterRate=0.0877048671245575}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 19:05:30,389 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 19:05:30,391 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=16, masterSystemTime=1689447930189 2023-07-15 19:05:30,393 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-15 19:05:30,393 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-15 19:05:30,394 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,45281,1689447925898, state=OPEN 2023-07-15 19:05:30,396 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 19:05:30,396 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 19:05:30,398 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-15 19:05:30,398 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,45281,1689447925898 in 360 msec 2023-07-15 19:05:30,401 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-15 19:05:30,401 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 518 msec 2023-07-15 19:05:30,401 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,43223,1689447925851 had 2 regions 2023-07-15 19:05:30,402 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,43223,1689447925851, splitWal=true, meta=true, isMeta: false 2023-07-15 19:05:30,405 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,43223,1689447925851 WAL count=1, meta=false 2023-07-15 19:05:30,405 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375}] 2023-07-15 19:05:30,407 DEBUG [PEWorker-1] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:30,408 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375, worker=jenkins-hbase4.apache.org,45281,1689447925898}] 2023-07-15 19:05:30,561 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45281] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=18 2023-07-15 19:05:30,574 DEBUG [hconnection-0x1e8cbdf7-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:43223 this server is in the failed servers list 2023-07-15 19:05:30,576 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375, size=1.1 K (1148bytes) 2023-07-15 19:05:30,576 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375 2023-07-15 19:05:30,577 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375 after 1ms 2023-07-15 19:05:30,580 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375 took 4ms 2023-07-15 19:05:30,583 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 44c8b933b39ea824f34757f7c29e20d7: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-15 19:05:30,585 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375 so closing down 2023-07-15 19:05:30,585 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 19:05:30,588 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375.temp 2023-07-15 19:05:30,589 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 19:05:30,600 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-15 19:05:30,602 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375.temp to hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/recovered.edits/0000000000000000013 2023-07-15 19:05:30,602 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 22 ms; skipped=1; WAL=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375, size=1.1 K, length=1148, corrupted=false, cancelled=false 2023-07-15 19:05:30,603 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375, journal: Splitting hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375, size=1.1 K (1148bytes) at 1689447930576Finishing writing output for hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375 so closing down at 1689447930585 (+9 ms)Creating recovered edits writer path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375.temp at 1689447930589 (+4 ms)3 split writer threads finished at 1689447930589Closed recovered edits writer path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689447930600 (+11 ms)Rename recovered edits hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375.temp to hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/recovered.edits/0000000000000000013 at 1689447930602 (+2 ms)Processed 11 edits across 1 Regions in 22 ms; skipped=1; WAL=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375, size=1.1 K, length=1148, corrupted=false, cancelled=false at 1689447930603 (+1 ms) 2023-07-15 19:05:30,603 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375 2023-07-15 19:05:30,603 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=18 2023-07-15 19:05:30,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38491] master.HMaster(3641): Remote procedure done, pid=18 2023-07-15 19:05:30,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38491] wal.WALSplitUtil(141): Moved hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting/jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375 to hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/oldWALs 2023-07-15 19:05:30,610 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-15 19:05:30,610 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375, worker=jenkins-hbase4.apache.org,45281,1689447925898 in 200 msec 2023-07-15 19:05:30,611 DEBUG [PEWorker-2] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:30,614 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=12 2023-07-15 19:05:30,614 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C43223%2C1689447925851.1689447927375, worker=jenkins-hbase4.apache.org,45281,1689447925898 in 206 msec 2023-07-15 19:05:30,616 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,43223,1689447925851-splitting dir is empty, no logs to split. 2023-07-15 19:05:30,617 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,43223,1689447925851 WAL count=0, meta=false 2023-07-15 19:05:30,617 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,43223,1689447925851 WAL splitting is done? wals=0, meta=false 2023-07-15 19:05:30,619 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=44c8b933b39ea824f34757f7c29e20d7, ASSIGN}] 2023-07-15 19:05:30,620 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=44c8b933b39ea824f34757f7c29e20d7, ASSIGN 2023-07-15 19:05:30,621 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=44c8b933b39ea824f34757f7c29e20d7, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 19:05:30,771 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=44c8b933b39ea824f34757f7c29e20d7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:30,772 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689447930771"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689447930771"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689447930771"}]},"ts":"1689447930771"} 2023-07-15 19:05:30,773 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:43223 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43223 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 19:05:30,774 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:43223 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43223 2023-07-15 19:05:30,780 DEBUG [hconnection-0x1e8cbdf7-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:43223 this server is in the failed servers list 2023-07-15 19:05:30,878 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE; OpenRegionProcedure 44c8b933b39ea824f34757f7c29e20d7, server=jenkins-hbase4.apache.org,45281,1689447925898}] 2023-07-15 19:05:31,037 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:31,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 44c8b933b39ea824f34757f7c29e20d7, NAME => 't,,1689447928585.44c8b933b39ea824f34757f7c29e20d7.', STARTKEY => '', ENDKEY => ''} 2023-07-15 19:05:31,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:31,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 19:05:31,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:31,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:31,039 INFO [StoreOpener-44c8b933b39ea824f34757f7c29e20d7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:31,041 DEBUG [StoreOpener-44c8b933b39ea824f34757f7c29e20d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/f 2023-07-15 19:05:31,041 DEBUG [StoreOpener-44c8b933b39ea824f34757f7c29e20d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/f 2023-07-15 19:05:31,041 INFO [StoreOpener-44c8b933b39ea824f34757f7c29e20d7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44c8b933b39ea824f34757f7c29e20d7 columnFamilyName f 2023-07-15 19:05:31,042 INFO [StoreOpener-44c8b933b39ea824f34757f7c29e20d7-1] regionserver.HStore(310): Store=44c8b933b39ea824f34757f7c29e20d7/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:31,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:31,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:31,047 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/recovered.edits/0000000000000000013 2023-07-15 19:05:31,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/recovered.edits/0000000000000000013 2023-07-15 19:05:31,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 44c8b933b39ea824f34757f7c29e20d7 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-15 19:05:31,068 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/.tmp/f/85e15a7bbc9c4badbee13fb7c8242a8d 2023-07-15 19:05:31,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/.tmp/f/85e15a7bbc9c4badbee13fb7c8242a8d as hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/f/85e15a7bbc9c4badbee13fb7c8242a8d 2023-07-15 19:05:31,086 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/f/85e15a7bbc9c4badbee13fb7c8242a8d, entries=10, sequenceid=13, filesize=4.8 K 2023-07-15 19:05:31,087 DEBUG [hconnection-0x1e8cbdf7-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:43223 this server is in the failed servers list 2023-07-15 19:05:31,087 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for 44c8b933b39ea824f34757f7c29e20d7 in 36ms, sequenceid=13, compaction requested=false; wal=null 2023-07-15 19:05:31,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-15 19:05:31,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/recovered.edits/0000000000000000013 2023-07-15 19:05:31,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:31,096 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-15 19:05:31,098 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 44c8b933b39ea824f34757f7c29e20d7; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71668084, jitterRate=0.06793767213821411}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 19:05:31,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 44c8b933b39ea824f34757f7c29e20d7: 2023-07-15 19:05:31,099 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7., pid=20, masterSystemTime=1689447931031 2023-07-15 19:05:31,101 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:31,101 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:31,102 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=44c8b933b39ea824f34757f7c29e20d7, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:31,102 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689447931102"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689447931102"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689447931102"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689447931102"}]},"ts":"1689447931102"} 2023-07-15 19:05:31,107 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-07-15 19:05:31,107 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; OpenRegionProcedure 44c8b933b39ea824f34757f7c29e20d7, server=jenkins-hbase4.apache.org,45281,1689447925898 in 227 msec 2023-07-15 19:05:31,109 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=12 2023-07-15 19:05:31,109 INFO [PEWorker-1] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,43223,1689447925851 after splitting done 2023-07-15 19:05:31,109 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=44c8b933b39ea824f34757f7c29e20d7, ASSIGN in 488 msec 2023-07-15 19:05:31,110 DEBUG [PEWorker-1] master.DeadServer(114): Removed jenkins-hbase4.apache.org,43223,1689447925851 from processing; numProcessing=0 2023-07-15 19:05:31,113 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,43223,1689447925851, splitWal=true, meta=true in 1.5500 sec 2023-07-15 19:05:31,600 DEBUG [Listener at localhost/42557] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c11e80b to 127.0.0.1:54942 2023-07-15 19:05:31,601 DEBUG [Listener at localhost/42557] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 19:05:31,601 INFO [Listener at localhost/42557] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-15 19:05:31,615 INFO [Listener at localhost/42557] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=268 (was 294), OpenFileDescriptor=569 (was 585), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=229 (was 223) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=6158 (was 6212) 2023-07-15 19:05:31,626 INFO [Listener at localhost/42557] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=268, OpenFileDescriptor=569, MaxFileDescriptor=60000, SystemLoadAverage=229, ProcessCount=170, AvailableMemoryMB=6157 2023-07-15 19:05:31,627 INFO [Listener at localhost/42557] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:43223 2023-07-15 19:05:31,639 INFO [Listener at localhost/42557] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 19:05:31,639 INFO [Listener at localhost/42557] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 19:05:31,639 INFO [Listener at localhost/42557] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 19:05:31,639 INFO [Listener at localhost/42557] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 19:05:31,640 INFO [Listener at localhost/42557] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 19:05:31,640 INFO [Listener at localhost/42557] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 19:05:31,640 INFO [Listener at localhost/42557] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 19:05:31,641 INFO [Listener at localhost/42557] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36163 2023-07-15 19:05:31,642 INFO [Listener at localhost/42557] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 19:05:31,642 DEBUG [Listener at localhost/42557] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 19:05:31,643 INFO [Listener at localhost/42557] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 19:05:31,644 INFO [Listener at localhost/42557] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 19:05:31,645 INFO [Listener at localhost/42557] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36163 connecting to ZooKeeper ensemble=127.0.0.1:54942 2023-07-15 19:05:31,648 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:361630x0, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 19:05:31,649 DEBUG [Listener at localhost/42557] zookeeper.ZKUtil(162): regionserver:361630x0, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 19:05:31,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:36163-0x1016a6003100008 connected 2023-07-15 19:05:31,650 DEBUG [Listener at localhost/42557] zookeeper.ZKUtil(162): regionserver:36163-0x1016a6003100008, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-15 19:05:31,652 DEBUG [Listener at localhost/42557] zookeeper.ZKUtil(164): regionserver:36163-0x1016a6003100008, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 19:05:31,652 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36163 2023-07-15 19:05:31,652 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36163 2023-07-15 19:05:31,652 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36163 2023-07-15 19:05:31,653 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36163 2023-07-15 19:05:31,653 DEBUG [Listener at localhost/42557] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36163 2023-07-15 19:05:31,655 INFO [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(951): ClusterId : 3d364618-7816-41f1-b41f-65e5960beece 2023-07-15 19:05:31,655 DEBUG [RS:2;jenkins-hbase4:36163] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 19:05:31,659 DEBUG [RS:2;jenkins-hbase4:36163] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 19:05:31,659 DEBUG [RS:2;jenkins-hbase4:36163] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 19:05:31,661 DEBUG [RS:2;jenkins-hbase4:36163] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 19:05:31,662 DEBUG [RS:2;jenkins-hbase4:36163] zookeeper.ReadOnlyZKClient(139): Connect 0x4ef37efe to 127.0.0.1:54942 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 19:05:31,666 DEBUG [RS:2;jenkins-hbase4:36163] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40349950, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 19:05:31,667 DEBUG [RS:2;jenkins-hbase4:36163] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@440b158e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 19:05:31,676 DEBUG [RS:2;jenkins-hbase4:36163] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:36163 2023-07-15 19:05:31,676 INFO [RS:2;jenkins-hbase4:36163] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 19:05:31,676 INFO [RS:2;jenkins-hbase4:36163] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 19:05:31,676 DEBUG [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 19:05:31,677 INFO [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,38491,1689447924569 with isa=jenkins-hbase4.apache.org/172.31.14.131:36163, startcode=1689447931638 2023-07-15 19:05:31,677 DEBUG [RS:2;jenkins-hbase4:36163] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 19:05:31,681 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54035, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 19:05:31,681 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38491] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:31,682 DEBUG [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37 2023-07-15 19:05:31,682 DEBUG [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:45815 2023-07-15 19:05:31,682 DEBUG [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-15 19:05:31,684 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 19:05:31,684 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 19:05:31,684 DEBUG [RS:2;jenkins-hbase4:36163] zookeeper.ZKUtil(162): regionserver:36163-0x1016a6003100008, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:31,684 WARN [RS:2;jenkins-hbase4:36163] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 19:05:31,684 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,36163,1689447931638] 2023-07-15 19:05:31,684 INFO [RS:2;jenkins-hbase4:36163] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 19:05:31,684 DEBUG [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(1948): logDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:31,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:31,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:31,689 DEBUG [RS:2;jenkins-hbase4:36163] zookeeper.ZKUtil(162): regionserver:36163-0x1016a6003100008, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:31,690 DEBUG [RS:2;jenkins-hbase4:36163] zookeeper.ZKUtil(162): regionserver:36163-0x1016a6003100008, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:31,691 DEBUG [RS:2;jenkins-hbase4:36163] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 19:05:31,691 INFO [RS:2;jenkins-hbase4:36163] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 19:05:31,696 INFO [RS:2;jenkins-hbase4:36163] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 19:05:31,697 INFO [RS:2;jenkins-hbase4:36163] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 19:05:31,697 INFO [RS:2;jenkins-hbase4:36163] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:31,697 INFO [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 19:05:31,698 INFO [RS:2;jenkins-hbase4:36163] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:31,699 DEBUG [RS:2;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:31,699 DEBUG [RS:2;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:31,699 DEBUG [RS:2;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:31,699 DEBUG [RS:2;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:31,699 DEBUG [RS:2;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:31,699 DEBUG [RS:2;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 19:05:31,699 DEBUG [RS:2;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:31,699 DEBUG [RS:2;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:31,699 DEBUG [RS:2;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:31,700 DEBUG [RS:2;jenkins-hbase4:36163] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 19:05:31,700 INFO [RS:2;jenkins-hbase4:36163] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:31,700 INFO [RS:2;jenkins-hbase4:36163] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:31,700 INFO [RS:2;jenkins-hbase4:36163] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:31,715 INFO [RS:2;jenkins-hbase4:36163] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 19:05:31,715 INFO [RS:2;jenkins-hbase4:36163] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36163,1689447931638-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 19:05:31,726 INFO [RS:2;jenkins-hbase4:36163] regionserver.Replication(203): jenkins-hbase4.apache.org,36163,1689447931638 started 2023-07-15 19:05:31,726 INFO [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,36163,1689447931638, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:36163, sessionid=0x1016a6003100008 2023-07-15 19:05:31,726 INFO [Listener at localhost/42557] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:36163,5,FailOnTimeoutGroup] 2023-07-15 19:05:31,726 DEBUG [RS:2;jenkins-hbase4:36163] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 19:05:31,726 INFO [Listener at localhost/42557] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,43223,1689447925851 2023-07-15 19:05:31,727 DEBUG [RS:2;jenkins-hbase4:36163] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:31,727 DEBUG [RS:2;jenkins-hbase4:36163] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36163,1689447931638' 2023-07-15 19:05:31,727 DEBUG [RS:2;jenkins-hbase4:36163] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 19:05:31,727 INFO [Listener at localhost/42557] client.TestScannerTimeout(177): START ************ test3686b 2023-07-15 19:05:31,727 DEBUG [RS:2;jenkins-hbase4:36163] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 19:05:31,728 DEBUG [RS:2;jenkins-hbase4:36163] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 19:05:31,728 DEBUG [RS:2;jenkins-hbase4:36163] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 19:05:31,728 DEBUG [RS:2;jenkins-hbase4:36163] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:31,728 DEBUG [RS:2;jenkins-hbase4:36163] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36163,1689447931638' 2023-07-15 19:05:31,728 DEBUG [RS:2;jenkins-hbase4:36163] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 19:05:31,728 WARN [RPCClient-NioEventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:43223 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43223 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 19:05:31,728 DEBUG [RS:2;jenkins-hbase4:36163] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 19:05:31,728 DEBUG [RPCClient-NioEventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:43223 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:43223 2023-07-15 19:05:31,730 DEBUG [RS:2;jenkins-hbase4:36163] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 19:05:31,730 INFO [RS:2;jenkins-hbase4:36163] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 19:05:31,730 INFO [RS:2;jenkins-hbase4:36163] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 19:05:31,831 DEBUG [hconnection-0x31531590-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 19:05:31,833 INFO [RS:2;jenkins-hbase4:36163] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36163%2C1689447931638, suffix=, logDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,36163,1689447931638, archiveDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/oldWALs, maxLogs=32 2023-07-15 19:05:31,834 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47428, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 19:05:31,836 DEBUG [Listener at localhost/42557] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-15 19:05:31,837 DEBUG [Listener at localhost/42557] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:31,837 DEBUG [Listener at localhost/42557] zookeeper.ReadOnlyZKClient(139): Connect 0x34b18741 to 127.0.0.1:54942 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 19:05:31,846 DEBUG [Listener at localhost/42557] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d5176de, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 19:05:31,849 DEBUG [hconnection-0x2bbe7d10-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 19:05:31,851 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47436, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 19:05:31,859 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41525,DS-0eeb57f1-6175-43f3-9611-c11b9699d2f2,DISK] 2023-07-15 19:05:31,859 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42117,DS-938bd891-cf5b-4388-85c8-f0af346e90f6,DISK] 2023-07-15 19:05:31,867 INFO [RS:2;jenkins-hbase4:36163] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,36163,1689447931638/jenkins-hbase4.apache.org%2C36163%2C1689447931638.1689447931835 2023-07-15 19:05:31,867 DEBUG [RS:2;jenkins-hbase4:36163] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41525,DS-0eeb57f1-6175-43f3-9611-c11b9699d2f2,DISK], DatanodeInfoWithStorage[127.0.0.1:42117,DS-938bd891-cf5b-4388-85c8-f0af346e90f6,DISK]] 2023-07-15 19:05:33,097 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-15 19:05:33,154 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-15 19:05:34,739 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 19:05:34,740 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-15 19:05:46,861 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease -946146433059258359 expired clientIPAndPort=172.31.14.131:47436, userName=jenkins, regionInfo=t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:48,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45281] regionserver.RSRpcServices(3094): Client tried to access missing scanner -946146433059258359 2023-07-15 19:05:48,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45281] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 172.31.14.131:47436 deadline: 1689448008862, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-946146433059258359'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-15 19:05:48,864 DEBUG [Listener at localhost/42557] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-946146433059258359'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-946146433059258359'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-946146433059258359'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-15 19:05:48,868 DEBUG [Listener at localhost/42557] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b18741 to 127.0.0.1:54942 2023-07-15 19:05:48,868 DEBUG [Listener at localhost/42557] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 19:05:48,868 INFO [Listener at localhost/42557] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-15 19:05:48,884 INFO [Listener at localhost/42557] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=296 (was 268) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=36163 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:54942@0x34b18741 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1423662552.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36163 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=36163 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:45815 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=36163 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:54942@0x4ef37efe-SendThread(127.0.0.1:54942) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/cluster_a9497dea-7554-f5e1-575e-fef856a8e7af/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36163 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-1710966795-172.31.14.131-1689447921569:blk_1073741845_1021, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-507150746_17 at /127.0.0.1:44664 [Receiving block BP-1710966795-172.31.14.131-1689447921569:blk_1073741845_1021] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36163 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36163 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS:2;jenkins-hbase4:36163-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-1710966795-172.31.14.131-1689447921569:blk_1073741845_1021, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-507150746_17 at /127.0.0.1:52812 [Receiving block BP-1710966795-172.31.14.131-1689447921569:blk_1073741845_1021] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37-prefix:jenkins-hbase4.apache.org,36163,1689447931638 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=36163 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/cluster_a9497dea-7554-f5e1-575e-fef856a8e7af/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:54942@0x4ef37efe-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:36163 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36163 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:54942@0x34b18741-SendThread(127.0.0.1:54942) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: jenkins-hbase4:36163Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36163 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:54942@0x4ef37efe sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1423662552.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:54942@0x34b18741-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) - Thread LEAK? -, OpenFileDescriptor=573 (was 569) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=179 (was 229), ProcessCount=170 (was 170), AvailableMemoryMB=6140 (was 6157) 2023-07-15 19:05:48,898 INFO [Listener at localhost/42557] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=296, OpenFileDescriptor=573, MaxFileDescriptor=60000, SystemLoadAverage=179, ProcessCount=170, AvailableMemoryMB=6140 2023-07-15 19:05:48,898 INFO [Listener at localhost/42557] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:43223 2023-07-15 19:05:48,899 INFO [Listener at localhost/42557] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,43223,1689447925851 2023-07-15 19:05:48,899 INFO [Listener at localhost/42557] client.TestScannerTimeout(104): START************ test2772 2023-07-15 19:05:48,914 DEBUG [Listener at localhost/42557] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-15 19:05:48,914 DEBUG [Listener at localhost/42557] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:48,915 DEBUG [Listener at localhost/42557] zookeeper.ReadOnlyZKClient(139): Connect 0x42d7d10c to 127.0.0.1:54942 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 19:05:48,921 DEBUG [Listener at localhost/42557] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a742b87, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 19:05:48,921 ERROR [Listener at localhost/42557] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,45281,1689447925898: die! ***** 2023-07-15 19:05:48,922 ERROR [Listener at localhost/42557] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-15 19:05:48,922 DEBUG [Listener at localhost/42557] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-15 19:05:48,923 DEBUG [Listener at localhost/42557] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-15 19:05:48,930 DEBUG [Listener at localhost/42557] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-15 19:05:48,932 DEBUG [Listener at localhost/42557] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-15 19:05:48,950 INFO [Listener at localhost/42557] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 988282880, "init": 513802240, "max": 2051014656, "used": 265920336 }, "NonHeapMemoryUsage": { "committed": 123887616, "init": 2555904, "max": -1, "used": 121478832 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 12870, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 4, "ProcessCallTime_num_ops": 63, "ProcessCallTime_min": 1, "ProcessCallTime_max": 2, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 2, "ProcessCallTime_75th_percentile": 2, "ProcessCallTime_90th_percentile": 2, "ProcessCallTime_95th_percentile": 2, "ProcessCallTime_98th_percentile": 2, "ProcessCallTime_99th_percentile": 2, "ProcessCallTime_99.9th_percentile": 2, "ProcessCallTime_TimeRangeCount_0-1": 1, "ProcessCallTime_TimeRangeCount_1-3": 1, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 63, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 63, "TotalCallTime_min": 1, "TotalCallTime_max": 2, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 2, "TotalCallTime_75th_percentile": 2, "TotalCallTime_90th_percentile": 2, "TotalCallTime_95th_percentile": 2, "TotalCallTime_98th_percentile": 2, "TotalCallTime_99th_percentile": 2, "TotalCallTime_99.9th_percentile": 2, "TotalCallTime_TimeRangeCount_0-1": 1, "TotalCallTime_TimeRangeCount_1-3": 1, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 63, "ResponseSize_min": 197, "ResponseSize_max": 679, "ResponseSize_mean": 438, "ResponseSize_25th_percentile": 199, "ResponseSize_median": 202, "ResponseSize_75th_percentile": 676, "ResponseSize_90th_percentile": 678, "ResponseSize_95th_percentile": 678, "ResponseSize_98th_percentile": 678, "ResponseSize_99th_percentile": 678, "ResponseSize_99.9th_percentile": 678, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 6, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 1, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 63, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 18485 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:54942", "tag.serverName": "jenkins-hbase4.apache.org,43223,1689447925851", "tag.clusterId": "3d364618-7816-41f1-b41f-65e5960beece", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 0, "storeCount": 0, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 0, "avgStoreFileAge": 0, "numReferenceFiles": 0, "regionServerStartTime": 1689447925851, "averageRegionSize": 0, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 1787, "localBytesRead": 1787, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 7076, "ByteBuffAllocatorPoolAllocationBytes": 665600, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 33, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 2, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 11, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 20, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 11, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 20, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 0, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 2, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 0, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 0, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 11, "ScanSize_min": 0, "ScanSize_max": 0, "ScanSize_mean": 0, "ScanSize_25th_percentile": 0, "ScanSize_median": 0, "ScanSize_75th_percentile": 0, "ScanSize_90th_percentile": 0, "ScanSize_95th_percentile": 0, "ScanSize_98th_percentile": 0, "ScanSize_99th_percentile": 0, "ScanSize_99.9th_percentile": 0, "slowGetCount": 0, "flushedMemstoreBytes": 0, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 0, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.480948374180283, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 24, "ServerWriteQueryPerSecond_mean_rate": 1.1026436689695978, "ServerWriteQueryPerSecond_1min_rate": 3.738243758742744, "ServerWriteQueryPerSecond_5min_rate": 4.565901237603426, "ServerWriteQueryPerSecond_15min_rate": 4.720662978343764 } ] } 2023-07-15 19:05:48,951 WARN [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=38491] master.MasterRpcServices(612): jenkins-hbase4.apache.org,45281,1689447925898 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,45281,1689447925898: die! ***** 2023-07-15 19:05:48,957 INFO [Listener at localhost/42557] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,45281,1689447925898' ***** 2023-07-15 19:05:48,958 INFO [Listener at localhost/42557] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-15 19:05:48,958 INFO [RS:1;jenkins-hbase4:45281] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 19:05:48,959 INFO [RS:1;jenkins-hbase4:45281] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-15 19:05:48,959 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 19:05:48,959 INFO [RS:1;jenkins-hbase4:45281] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-15 19:05:48,960 INFO [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(3305): Received CLOSE for 24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:48,961 INFO [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(3305): Received CLOSE for 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:48,961 INFO [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:48,961 DEBUG [RS:1;jenkins-hbase4:45281] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x01bd7640 to 127.0.0.1:54942 2023-07-15 19:05:48,961 DEBUG [RS:1;jenkins-hbase4:45281] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 19:05:48,961 INFO [RS:1;jenkins-hbase4:45281] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-15 19:05:48,961 INFO [RS:1;jenkins-hbase4:45281] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-15 19:05:48,961 INFO [RS:1;jenkins-hbase4:45281] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-15 19:05:48,961 INFO [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-15 19:05:48,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 24f2c4350af36d0c0546d743c16c75c0, disabling compactions & flushes 2023-07-15 19:05:48,964 INFO [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-15 19:05:48,964 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:48,964 DEBUG [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(1478): Online Regions={24f2c4350af36d0c0546d743c16c75c0=hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0., 44c8b933b39ea824f34757f7c29e20d7=t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7., 1588230740=hbase:meta,,1.1588230740} 2023-07-15 19:05:48,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:48,967 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 19:05:48,967 DEBUG [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(1504): Waiting on 1588230740, 24f2c4350af36d0c0546d743c16c75c0, 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:48,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. after waiting 0 ms 2023-07-15 19:05:48,968 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 19:05:48,968 DEBUG [hconnection-0x5e93565a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 19:05:48,968 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 19:05:48,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:48,968 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 19:05:48,968 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 19:05:48,979 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:48,980 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:48,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 24f2c4350af36d0c0546d743c16c75c0: 2023-07-15 19:05:48,980 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54730, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 19:05:48,982 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=45281] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45281,1689447925898 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 19:05:48,982 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=45281] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:54730 deadline: 1689448008981, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45281,1689447925898 aborting 2023-07-15 19:05:48,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:48,986 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 44c8b933b39ea824f34757f7c29e20d7, disabling compactions & flushes 2023-07-15 19:05:48,988 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:48,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:48,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. after waiting 0 ms 2023-07-15 19:05:48,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:48,994 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 834 in region hbase:meta,,1.1588230740 2023-07-15 19:05:48,994 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 19:05:48,995 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 19:05:48,996 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 19:05:48,997 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:48,997 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-15 19:05:48,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 44c8b933b39ea824f34757f7c29e20d7: 2023-07-15 19:05:48,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:49,089 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45281] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45281,1689447925898 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 19:05:49,089 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45281] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:54730 deadline: 1689448009089, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,45281,1689447925898 aborting 2023-07-15 19:05:49,168 INFO [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,45281,1689447925898; all regions closed. 2023-07-15 19:05:49,180 DEBUG [RS:1;jenkins-hbase4:45281] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 19:05:49,180 INFO [RS:1;jenkins-hbase4:45281] regionserver.LeaseManager(133): Closed leases 2023-07-15 19:05:49,181 INFO [RS:1;jenkins-hbase4:45281] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-15 19:05:49,181 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 19:05:49,182 INFO [RS:1;jenkins-hbase4:45281] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45281 2023-07-15 19:05:49,185 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:49,185 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 19:05:49,185 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:36163-0x1016a6003100008, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,45281,1689447925898 2023-07-15 19:05:49,185 ERROR [Listener at localhost/42557-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1c83c0ae rejected from java.util.concurrent.ThreadPoolExecutor@18bc47e6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:603) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-07-15 19:05:49,185 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:36163-0x1016a6003100008, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 19:05:49,186 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 19:05:49,187 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,45281,1689447925898] 2023-07-15 19:05:49,187 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,45281,1689447925898; numProcessing=1 2023-07-15 19:05:49,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36163-0x1016a6003100008, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:49,188 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,45281,1689447925898 znode expired, triggering replicatorRemoved event 2023-07-15 19:05:49,189 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,45281,1689447925898 already deleted, retry=false 2023-07-15 19:05:49,189 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,45281,1689447925898 on jenkins-hbase4.apache.org,38491,1689447924569 2023-07-15 19:05:49,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36163-0x1016a6003100008, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:49,190 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=21, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,45281,1689447925898, splitWal=true, meta=true 2023-07-15 19:05:49,190 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=21 for jenkins-hbase4.apache.org,45281,1689447925898 (carryingMeta=true) jenkins-hbase4.apache.org,45281,1689447925898/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@41e92ef5[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-15 19:05:49,192 INFO [PEWorker-3] procedure.ServerCrashProcedure(161): Start pid=21, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,45281,1689447925898, splitWal=true, meta=true 2023-07-15 19:05:49,193 INFO [PEWorker-3] procedure.ServerCrashProcedure(300): Splitting WALs pid=21, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,45281,1689447925898, splitWal=true, meta=true, isMeta: true 2023-07-15 19:05:49,197 DEBUG [PEWorker-3] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting 2023-07-15 19:05:49,198 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,45281,1689447925898 WAL count=1, meta=true 2023-07-15 19:05:49,198 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta}] 2023-07-15 19:05:49,201 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:49,201 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta, worker=jenkins-hbase4.apache.org,36163,1689447931638}] 2023-07-15 19:05:49,287 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 19:05:49,287 INFO [RS:1;jenkins-hbase4:45281] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,45281,1689447925898; zookeeper connection closed. 2023-07-15 19:05:49,287 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:45281-0x1016a6003100002, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 19:05:49,287 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2256da45] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2256da45 2023-07-15 19:05:49,292 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:45281 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45281 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 19:05:49,293 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:45281 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45281 2023-07-15 19:05:49,354 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:49,354 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 19:05:49,355 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56620, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 19:05:49,356 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36163] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=23 2023-07-15 19:05:49,371 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta, size=1.3 K (1341bytes) 2023-07-15 19:05:49,371 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta 2023-07-15 19:05:49,372 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta after 1ms 2023-07-15 19:05:49,377 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta took 6ms 2023-07-15 19:05:49,381 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 13 store_sequence_id { family_name: "info" sequence_id: 15 } store_sequence_id { family_name: "rep_barrier" sequence_id: 13 } store_sequence_id { family_name: "table" sequence_id: 13 } 2023-07-15 19:05:49,382 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta so closing down 2023-07-15 19:05:49,382 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 19:05:49,386 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta.temp 2023-07-15 19:05:49,386 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 19:05:49,399 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-15 19:05:49,401 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta.temp to hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-15 19:05:49,401 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 3 edits across 1 Regions in 23 ms; skipped=1; WAL=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta, size=1.3 K, length=1341, corrupted=false, cancelled=false 2023-07-15 19:05:49,401 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta, journal: Splitting hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta, size=1.3 K (1341bytes) at 1689447949371Finishing writing output for hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta so closing down at 1689447949382 (+11 ms)Creating recovered edits writer path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta.temp at 1689447949386 (+4 ms)3 split writer threads finished at 1689447949386Closed recovered edits writer path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689447949399 (+13 ms)Rename recovered edits hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta.temp to hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 at 1689447949401 (+2 ms)Processed 3 edits across 1 Regions in 23 ms; skipped=1; WAL=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta, size=1.3 K, length=1341, corrupted=false, cancelled=false at 1689447949401 2023-07-15 19:05:49,401 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta 2023-07-15 19:05:49,401 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=23 2023-07-15 19:05:49,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38491] master.HMaster(3641): Remote procedure done, pid=23 2023-07-15 19:05:49,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38491] wal.WALSplitUtil(141): Moved hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta to hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/oldWALs 2023-07-15 19:05:49,408 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-15 19:05:49,408 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta, worker=jenkins-hbase4.apache.org,36163,1689447931638 in 205 msec 2023-07-15 19:05:49,410 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:49,412 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-07-15 19:05:49,412 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C45281%2C1689447925898.meta.1689447930199.meta, worker=jenkins-hbase4.apache.org,36163,1689447931638 in 212 msec 2023-07-15 19:05:49,414 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting dir is empty, no logs to split. 2023-07-15 19:05:49,414 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,45281,1689447925898 WAL count=0, meta=true 2023-07-15 19:05:49,414 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,45281,1689447925898 WAL splitting is done? wals=0, meta=true 2023-07-15 19:05:49,415 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-15 19:05:49,416 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-15 19:05:49,417 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 19:05:49,438 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 19:05:49,567 DEBUG [jenkins-hbase4:38491] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-15 19:05:49,567 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36163,1689447931638, state=OPENING 2023-07-15 19:05:49,571 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 19:05:49,571 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36163,1689447931638}] 2023-07-15 19:05:49,572 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 19:05:49,595 WARN [ReadOnlyZKClient-127.0.0.1:54942@0x42d7d10c] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-15 19:05:49,595 DEBUG [hconnection-0x5e93565a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 19:05:49,598 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56624, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 19:05:49,598 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36163] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:56624 deadline: 1689448009598, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:49,730 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-15 19:05:49,730 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 19:05:49,733 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36163%2C1689447931638.meta, suffix=.meta, logDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,36163,1689447931638, archiveDir=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/oldWALs, maxLogs=32 2023-07-15 19:05:49,749 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42117,DS-938bd891-cf5b-4388-85c8-f0af346e90f6,DISK] 2023-07-15 19:05:49,749 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41525,DS-0eeb57f1-6175-43f3-9611-c11b9699d2f2,DISK] 2023-07-15 19:05:49,751 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,36163,1689447931638/jenkins-hbase4.apache.org%2C36163%2C1689447931638.meta.1689447949733.meta 2023-07-15 19:05:49,751 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41525,DS-0eeb57f1-6175-43f3-9611-c11b9699d2f2,DISK], DatanodeInfoWithStorage[127.0.0.1:42117,DS-938bd891-cf5b-4388-85c8-f0af346e90f6,DISK]] 2023-07-15 19:05:49,751 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-15 19:05:49,752 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 19:05:49,752 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-15 19:05:49,752 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-15 19:05:49,752 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-15 19:05:49,752 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 19:05:49,752 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-15 19:05:49,752 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-15 19:05:49,754 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 19:05:49,756 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/info 2023-07-15 19:05:49,756 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/info 2023-07-15 19:05:49,756 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 19:05:49,764 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/info/54303d9474584fa398c46f9c8226c151 2023-07-15 19:05:49,765 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:49,765 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 19:05:49,766 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/rep_barrier 2023-07-15 19:05:49,766 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/rep_barrier 2023-07-15 19:05:49,766 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 19:05:49,767 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:49,767 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 19:05:49,768 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/table 2023-07-15 19:05:49,769 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/table 2023-07-15 19:05:49,770 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 19:05:49,779 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/table/4c97bbfc44c247bda10817d8b84d47e0 2023-07-15 19:05:49,779 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:49,780 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740 2023-07-15 19:05:49,782 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740 2023-07-15 19:05:49,783 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-15 19:05:49,788 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 8, skipped 0, firstSequenceIdInLog=16, maxSequenceIdInLog=17, path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-15 19:05:49,788 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=834 B heapSize=2.16 KB 2023-07-15 19:05:49,801 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=834 B at sequenceid=17 (bloomFilter=false), to=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/.tmp/info/7adb5f54ad4f422aa6cd743b1fdb0bd6 2023-07-15 19:05:49,807 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/.tmp/info/7adb5f54ad4f422aa6cd743b1fdb0bd6 as hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/info/7adb5f54ad4f422aa6cd743b1fdb0bd6 2023-07-15 19:05:49,813 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/info/7adb5f54ad4f422aa6cd743b1fdb0bd6, entries=8, sequenceid=17, filesize=5.5 K 2023-07-15 19:05:49,813 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~834 B/834, heapSize ~1.64 KB/1680, currentSize=0 B/0 for 1588230740 in 25ms, sequenceid=17, compaction requested=false; wal=null 2023-07-15 19:05:49,814 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-15 19:05:49,817 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 19:05:49,818 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 19:05:49,821 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=13 2023-07-15 19:05:49,822 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=18; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71453792, jitterRate=0.06474447250366211}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 19:05:49,822 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 19:05:49,823 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=25, masterSystemTime=1689447949724 2023-07-15 19:05:49,825 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-15 19:05:49,826 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-15 19:05:49,826 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36163,1689447931638, state=OPEN 2023-07-15 19:05:49,827 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 19:05:49,827 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 19:05:49,830 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-15 19:05:49,830 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36163,1689447931638 in 256 msec 2023-07-15 19:05:49,833 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=21 2023-07-15 19:05:49,833 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,45281,1689447925898 had 3 regions 2023-07-15 19:05:49,833 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 415 msec 2023-07-15 19:05:49,835 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=21, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,45281,1689447925898, splitWal=true, meta=true, isMeta: false 2023-07-15 19:05:49,837 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,45281,1689447925898 WAL count=1, meta=false 2023-07-15 19:05:49,837 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=21, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375}] 2023-07-15 19:05:49,839 DEBUG [PEWorker-2] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:49,840 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375, worker=jenkins-hbase4.apache.org,36163,1689447931638}] 2023-07-15 19:05:49,993 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36163] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=27 2023-07-15 19:05:50,007 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375, size=899 (899bytes) 2023-07-15 19:05:50,007 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375 2023-07-15 19:05:50,007 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375 after 0ms 2023-07-15 19:05:50,010 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375 took 4ms 2023-07-15 19:05:50,014 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 24f2c4350af36d0c0546d743c16c75c0: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-15 19:05:50,016 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 44c8b933b39ea824f34757f7c29e20d7: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-15 19:05:50,016 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375 so closing down 2023-07-15 19:05:50,016 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 19:05:50,018 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375.temp 2023-07-15 19:05:50,018 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 19:05:50,033 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-15 19:05:50,035 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375.temp to hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/recovered.edits/0000000000000000005 2023-07-15 19:05:50,035 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 4 edits across 1 Regions in 24 ms; skipped=2; WAL=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375, size=899, length=899, corrupted=false, cancelled=false 2023-07-15 19:05:50,036 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375, journal: Splitting hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375, size=899 (899bytes) at 1689447950007Finishing writing output for hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375 so closing down at 1689447950016 (+9 ms)Creating recovered edits writer path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375.temp at 1689447950018 (+2 ms)3 split writer threads finished at 1689447950018Closed recovered edits writer path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689447950034 (+16 ms)Rename recovered edits hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375.temp to hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/recovered.edits/0000000000000000005 at 1689447950035 (+1 ms)Processed 4 edits across 1 Regions in 24 ms; skipped=2; WAL=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375, size=899, length=899, corrupted=false, cancelled=false at 1689447950035 2023-07-15 19:05:50,036 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375 2023-07-15 19:05:50,036 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=27 2023-07-15 19:05:50,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38491] master.HMaster(3641): Remote procedure done, pid=27 2023-07-15 19:05:50,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38491] wal.WALSplitUtil(141): Moved hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting/jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375 to hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/oldWALs 2023-07-15 19:05:50,043 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=26 2023-07-15 19:05:50,043 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375, worker=jenkins-hbase4.apache.org,36163,1689447931638 in 200 msec 2023-07-15 19:05:50,044 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:50,046 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=21 2023-07-15 19:05:50,046 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=21, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C45281%2C1689447925898.1689447927375, worker=jenkins-hbase4.apache.org,36163,1689447931638 in 207 msec 2023-07-15 19:05:50,048 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/WALs/jenkins-hbase4.apache.org,45281,1689447925898-splitting dir is empty, no logs to split. 2023-07-15 19:05:50,048 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,45281,1689447925898 WAL count=0, meta=false 2023-07-15 19:05:50,048 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,45281,1689447925898 WAL splitting is done? wals=0, meta=false 2023-07-15 19:05:50,050 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=44c8b933b39ea824f34757f7c29e20d7, ASSIGN}, {pid=29, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=24f2c4350af36d0c0546d743c16c75c0, ASSIGN}] 2023-07-15 19:05:50,051 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=24f2c4350af36d0c0546d743c16c75c0, ASSIGN 2023-07-15 19:05:50,052 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=44c8b933b39ea824f34757f7c29e20d7, ASSIGN 2023-07-15 19:05:50,053 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=28, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=44c8b933b39ea824f34757f7c29e20d7, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 19:05:50,053 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=24f2c4350af36d0c0546d743c16c75c0, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 19:05:50,053 DEBUG [jenkins-hbase4:38491] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-15 19:05:50,054 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=44c8b933b39ea824f34757f7c29e20d7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:50,054 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=24f2c4350af36d0c0546d743c16c75c0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:50,054 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689447950054"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689447950054"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689447950054"}]},"ts":"1689447950054"} 2023-07-15 19:05:50,054 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689447950054"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689447950054"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689447950054"}]},"ts":"1689447950054"} 2023-07-15 19:05:50,055 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:45281 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45281 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 19:05:50,056 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:45281 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45281 2023-07-15 19:05:50,111 DEBUG [hconnection-0x5e93565a-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:45281 this server is in the failed servers list 2023-07-15 19:05:50,157 DEBUG [PEWorker-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 19:05:50,159 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 19:05:50,161 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 24f2c4350af36d0c0546d743c16c75c0, server=jenkins-hbase4.apache.org,36163,1689447931638}] 2023-07-15 19:05:50,163 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=28, state=RUNNABLE; OpenRegionProcedure 44c8b933b39ea824f34757f7c29e20d7, server=jenkins-hbase4.apache.org,36163,1689447931638}] 2023-07-15 19:05:50,219 DEBUG [hconnection-0x5e93565a-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:45281 this server is in the failed servers list 2023-07-15 19:05:50,318 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:50,319 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 24f2c4350af36d0c0546d743c16c75c0, NAME => 'hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0.', STARTKEY => '', ENDKEY => ''} 2023-07-15 19:05:50,319 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:50,319 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 19:05:50,319 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:50,319 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:50,321 INFO [StoreOpener-24f2c4350af36d0c0546d743c16c75c0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:50,322 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:50,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 44c8b933b39ea824f34757f7c29e20d7, NAME => 't,,1689447928585.44c8b933b39ea824f34757f7c29e20d7.', STARTKEY => '', ENDKEY => ''} 2023-07-15 19:05:50,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:50,322 DEBUG [StoreOpener-24f2c4350af36d0c0546d743c16c75c0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/info 2023-07-15 19:05:50,322 DEBUG [StoreOpener-24f2c4350af36d0c0546d743c16c75c0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/info 2023-07-15 19:05:50,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 19:05:50,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:50,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:50,323 INFO [StoreOpener-24f2c4350af36d0c0546d743c16c75c0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 24f2c4350af36d0c0546d743c16c75c0 columnFamilyName info 2023-07-15 19:05:50,323 INFO [StoreOpener-24f2c4350af36d0c0546d743c16c75c0-1] regionserver.HStore(310): Store=24f2c4350af36d0c0546d743c16c75c0/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:50,324 INFO [StoreOpener-44c8b933b39ea824f34757f7c29e20d7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:50,325 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:50,326 DEBUG [StoreOpener-44c8b933b39ea824f34757f7c29e20d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/f 2023-07-15 19:05:50,326 DEBUG [StoreOpener-44c8b933b39ea824f34757f7c29e20d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/f 2023-07-15 19:05:50,326 INFO [StoreOpener-44c8b933b39ea824f34757f7c29e20d7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44c8b933b39ea824f34757f7c29e20d7 columnFamilyName f 2023-07-15 19:05:50,327 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:50,328 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/recovered.edits/0000000000000000005 2023-07-15 19:05:50,331 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/recovered.edits/0000000000000000005 2023-07-15 19:05:50,332 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 24f2c4350af36d0c0546d743c16c75c0 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-15 19:05:50,333 DEBUG [StoreOpener-44c8b933b39ea824f34757f7c29e20d7-1] regionserver.HStore(539): loaded hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/f/85e15a7bbc9c4badbee13fb7c8242a8d 2023-07-15 19:05:50,334 INFO [StoreOpener-44c8b933b39ea824f34757f7c29e20d7-1] regionserver.HStore(310): Store=44c8b933b39ea824f34757f7c29e20d7/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 19:05:50,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:50,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:50,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:50,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-15 19:05:50,344 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 44c8b933b39ea824f34757f7c29e20d7; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74663510, jitterRate=0.11257299780845642}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 19:05:50,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 44c8b933b39ea824f34757f7c29e20d7: 2023-07-15 19:05:50,345 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7., pid=31, masterSystemTime=1689447950314 2023-07-15 19:05:50,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:50,351 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:50,352 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=44c8b933b39ea824f34757f7c29e20d7, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:50,352 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/.tmp/info/69b546d2e6fa4adfb5c09708477a4663 2023-07-15 19:05:50,352 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689447950352"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689447950352"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689447950352"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689447950352"}]},"ts":"1689447950352"} 2023-07-15 19:05:50,357 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=28 2023-07-15 19:05:50,357 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=28, state=SUCCESS; OpenRegionProcedure 44c8b933b39ea824f34757f7c29e20d7, server=jenkins-hbase4.apache.org,36163,1689447931638 in 191 msec 2023-07-15 19:05:50,358 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=t, region=44c8b933b39ea824f34757f7c29e20d7, ASSIGN in 307 msec 2023-07-15 19:05:50,361 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/.tmp/info/69b546d2e6fa4adfb5c09708477a4663 as hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/info/69b546d2e6fa4adfb5c09708477a4663 2023-07-15 19:05:50,368 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/info/69b546d2e6fa4adfb5c09708477a4663, entries=2, sequenceid=5, filesize=4.8 K 2023-07-15 19:05:50,369 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 24f2c4350af36d0c0546d743c16c75c0 in 37ms, sequenceid=5, compaction requested=false; wal=null 2023-07-15 19:05:50,370 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/recovered.edits/0000000000000000005 2023-07-15 19:05:50,373 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:50,375 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-15 19:05:50,377 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 24f2c4350af36d0c0546d743c16c75c0; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61106694, jitterRate=-0.08943930268287659}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 19:05:50,377 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 24f2c4350af36d0c0546d743c16c75c0: 2023-07-15 19:05:50,378 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0., pid=30, masterSystemTime=1689447950314 2023-07-15 19:05:50,379 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:50,379 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:50,380 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=24f2c4350af36d0c0546d743c16c75c0, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:50,380 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689447950380"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689447950380"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689447950380"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689447950380"}]},"ts":"1689447950380"} 2023-07-15 19:05:50,384 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-07-15 19:05:50,384 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 24f2c4350af36d0c0546d743c16c75c0, server=jenkins-hbase4.apache.org,36163,1689447931638 in 221 msec 2023-07-15 19:05:50,386 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=21 2023-07-15 19:05:50,387 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,45281,1689447925898 after splitting done 2023-07-15 19:05:50,387 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=24f2c4350af36d0c0546d743c16c75c0, ASSIGN in 334 msec 2023-07-15 19:05:50,387 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase4.apache.org,45281,1689447925898 from processing; numProcessing=0 2023-07-15 19:05:50,389 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,45281,1689447925898, splitWal=true, meta=true in 1.1980 sec 2023-07-15 19:05:50,430 DEBUG [Listener at localhost/42557] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x42d7d10c to 127.0.0.1:54942 2023-07-15 19:05:50,430 DEBUG [Listener at localhost/42557] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 19:05:50,430 INFO [Listener at localhost/42557] client.TestScannerTimeout(124): END ************ test2772 2023-07-15 19:05:50,443 INFO [Listener at localhost/42557] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=277 (was 296), OpenFileDescriptor=583 (was 573) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=164 (was 179), ProcessCount=170 (was 170), AvailableMemoryMB=6120 (was 6140) 2023-07-15 19:05:50,444 INFO [Listener at localhost/42557] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-15 19:05:50,444 INFO [Listener at localhost/42557] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-15 19:05:50,444 DEBUG [Listener at localhost/42557] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04ca72c7 to 127.0.0.1:54942 2023-07-15 19:05:50,444 DEBUG [Listener at localhost/42557] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 19:05:50,444 DEBUG [Listener at localhost/42557] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-15 19:05:50,444 DEBUG [Listener at localhost/42557] util.JVMClusterUtil(257): Found active master hash=1275026513, stopped=false 2023-07-15 19:05:50,444 INFO [Listener at localhost/42557] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,38491,1689447924569 2023-07-15 19:05:50,447 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:36163-0x1016a6003100008, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-15 19:05:50,447 INFO [Listener at localhost/42557] procedure2.ProcedureExecutor(629): Stopping 2023-07-15 19:05:50,447 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-15 19:05:50,447 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 19:05:50,447 DEBUG [Listener at localhost/42557] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0c1f39f4 to 127.0.0.1:54942 2023-07-15 19:05:50,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36163-0x1016a6003100008, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 19:05:50,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 19:05:50,448 DEBUG [Listener at localhost/42557] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 19:05:50,448 INFO [Listener at localhost/42557] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,36163,1689447931638' ***** 2023-07-15 19:05:50,448 INFO [Listener at localhost/42557] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-15 19:05:50,448 INFO [RS:2;jenkins-hbase4:36163] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 19:05:50,448 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 19:05:50,448 INFO [RS:2;jenkins-hbase4:36163] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-15 19:05:50,449 INFO [RS:2;jenkins-hbase4:36163] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-15 19:05:50,449 INFO [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(3305): Received CLOSE for 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:50,449 INFO [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(3305): Received CLOSE for 24f2c4350af36d0c0546d743c16c75c0 2023-07-15 19:05:50,449 INFO [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:50,449 DEBUG [RS:2;jenkins-hbase4:36163] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ef37efe to 127.0.0.1:54942 2023-07-15 19:05:50,449 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 44c8b933b39ea824f34757f7c29e20d7, disabling compactions & flushes 2023-07-15 19:05:50,449 DEBUG [RS:2;jenkins-hbase4:36163] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 19:05:50,449 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:50,449 INFO [RS:2;jenkins-hbase4:36163] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-15 19:05:50,450 INFO [RS:2;jenkins-hbase4:36163] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-15 19:05:50,450 INFO [RS:2;jenkins-hbase4:36163] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-15 19:05:50,449 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:50,450 INFO [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-15 19:05:50,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. after waiting 0 ms 2023-07-15 19:05:50,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:50,450 INFO [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-15 19:05:50,450 DEBUG [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(1478): Online Regions={44c8b933b39ea824f34757f7c29e20d7=t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7., 24f2c4350af36d0c0546d743c16c75c0=hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0., 1588230740=hbase:meta,,1.1588230740} 2023-07-15 19:05:50,450 DEBUG [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(1504): Waiting on 1588230740, 24f2c4350af36d0c0546d743c16c75c0, 44c8b933b39ea824f34757f7c29e20d7 2023-07-15 19:05:50,450 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 19:05:50,450 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 19:05:50,450 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 19:05:50,450 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 19:05:50,450 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 19:05:50,451 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-15 19:05:50,456 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/default/t/44c8b933b39ea824f34757f7c29e20d7/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-15 19:05:50,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:50,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 44c8b933b39ea824f34757f7c29e20d7: 2023-07-15 19:05:50,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689447928585.44c8b933b39ea824f34757f7c29e20d7. 2023-07-15 19:05:50,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 24f2c4350af36d0c0546d743c16c75c0, disabling compactions & flushes 2023-07-15 19:05:50,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:50,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:50,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. after waiting 0 ms 2023-07-15 19:05:50,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:50,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/namespace/24f2c4350af36d0c0546d743c16c75c0/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=5 2023-07-15 19:05:50,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:50,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 24f2c4350af36d0c0546d743c16c75c0: 2023-07-15 19:05:50,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689447927778.24f2c4350af36d0c0546d743c16c75c0. 2023-07-15 19:05:50,475 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/.tmp/info/dbb787cf5a5a44a9b9914447ffb4e0df 2023-07-15 19:05:50,483 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/.tmp/info/dbb787cf5a5a44a9b9914447ffb4e0df as hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/info/dbb787cf5a5a44a9b9914447ffb4e0df 2023-07-15 19:05:50,489 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/info/dbb787cf5a5a44a9b9914447ffb4e0df, entries=16, sequenceid=24, filesize=6.4 K 2023-07-15 19:05:50,490 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 39ms, sequenceid=24, compaction requested=true 2023-07-15 19:05:50,498 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=17 2023-07-15 19:05:50,498 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 19:05:50,498 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 19:05:50,498 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 19:05:50,498 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-15 19:05:50,650 INFO [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,36163,1689447931638; all regions closed. 2023-07-15 19:05:50,657 DEBUG [RS:2;jenkins-hbase4:36163] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/oldWALs 2023-07-15 19:05:50,657 INFO [RS:2;jenkins-hbase4:36163] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36163%2C1689447931638.meta:.meta(num 1689447949733) 2023-07-15 19:05:50,662 DEBUG [RS:2;jenkins-hbase4:36163] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/oldWALs 2023-07-15 19:05:50,662 INFO [RS:2;jenkins-hbase4:36163] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36163%2C1689447931638:(num 1689447931835) 2023-07-15 19:05:50,662 DEBUG [RS:2;jenkins-hbase4:36163] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 19:05:50,662 INFO [RS:2;jenkins-hbase4:36163] regionserver.LeaseManager(133): Closed leases 2023-07-15 19:05:50,662 INFO [RS:2;jenkins-hbase4:36163] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-15 19:05:50,662 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 19:05:50,663 INFO [RS:2;jenkins-hbase4:36163] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36163 2023-07-15 19:05:50,665 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 19:05:50,665 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:36163-0x1016a6003100008, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36163,1689447931638 2023-07-15 19:05:50,665 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:36163-0x1016a6003100008, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 19:05:50,666 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,36163,1689447931638] 2023-07-15 19:05:50,666 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,36163,1689447931638; numProcessing=1 2023-07-15 19:05:50,667 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,36163,1689447931638 already deleted, retry=false 2023-07-15 19:05:50,667 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,36163,1689447931638 expired; onlineServers=0 2023-07-15 19:05:50,667 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,38491,1689447924569' ***** 2023-07-15 19:05:50,667 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-15 19:05:50,668 DEBUG [M:0;jenkins-hbase4:38491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ac2566a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 19:05:50,668 INFO [M:0;jenkins-hbase4:38491] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,38491,1689447924569 2023-07-15 19:05:50,668 INFO [M:0;jenkins-hbase4:38491] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,38491,1689447924569; all regions closed. 2023-07-15 19:05:50,668 DEBUG [M:0;jenkins-hbase4:38491] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 19:05:50,668 DEBUG [M:0;jenkins-hbase4:38491] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-15 19:05:50,668 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-15 19:05:50,668 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689447926996] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689447926996,5,FailOnTimeoutGroup] 2023-07-15 19:05:50,668 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689447926997] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689447926997,5,FailOnTimeoutGroup] 2023-07-15 19:05:50,668 DEBUG [M:0;jenkins-hbase4:38491] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-15 19:05:50,669 INFO [M:0;jenkins-hbase4:38491] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-15 19:05:50,669 INFO [M:0;jenkins-hbase4:38491] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-15 19:05:50,669 INFO [M:0;jenkins-hbase4:38491] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-15 19:05:50,669 DEBUG [M:0;jenkins-hbase4:38491] master.HMaster(1512): Stopping service threads 2023-07-15 19:05:50,669 INFO [M:0;jenkins-hbase4:38491] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-15 19:05:50,669 ERROR [M:0;jenkins-hbase4:38491] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (114716313) connection to localhost/127.0.0.1:45815 from jenkins,5,PEWorkerGroup] 2023-07-15 19:05:50,670 INFO [M:0;jenkins-hbase4:38491] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-15 19:05:50,670 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-15 19:05:50,671 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-15 19:05:50,672 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 19:05:50,672 DEBUG [M:0;jenkins-hbase4:38491] zookeeper.ZKUtil(398): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-15 19:05:50,672 WARN [M:0;jenkins-hbase4:38491] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-15 19:05:50,672 INFO [M:0;jenkins-hbase4:38491] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-15 19:05:50,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 19:05:50,672 INFO [M:0;jenkins-hbase4:38491] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-15 19:05:50,672 DEBUG [M:0;jenkins-hbase4:38491] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-15 19:05:50,672 INFO [M:0;jenkins-hbase4:38491] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 19:05:50,672 DEBUG [M:0;jenkins-hbase4:38491] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 19:05:50,672 DEBUG [M:0;jenkins-hbase4:38491] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-15 19:05:50,673 DEBUG [M:0;jenkins-hbase4:38491] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 19:05:50,673 INFO [M:0;jenkins-hbase4:38491] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=105.75 KB heapSize=127.45 KB 2023-07-15 19:05:50,688 INFO [M:0;jenkins-hbase4:38491] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=105.75 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/1ef2b4eda62a44a3baa59009ee264918 2023-07-15 19:05:50,696 DEBUG [M:0;jenkins-hbase4:38491] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/1ef2b4eda62a44a3baa59009ee264918 as hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/1ef2b4eda62a44a3baa59009ee264918 2023-07-15 19:05:50,702 INFO [M:0;jenkins-hbase4:38491] regionserver.HStore(1080): Added hdfs://localhost:45815/user/jenkins/test-data/34c714f2-499c-ec89-56b4-6ca6af9f7d37/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/1ef2b4eda62a44a3baa59009ee264918, entries=31, sequenceid=244, filesize=8.4 K 2023-07-15 19:05:50,703 INFO [M:0;jenkins-hbase4:38491] regionserver.HRegion(2948): Finished flush of dataSize ~105.75 KB/108290, heapSize ~127.44 KB/130496, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 30ms, sequenceid=244, compaction requested=false 2023-07-15 19:05:50,705 INFO [M:0;jenkins-hbase4:38491] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 19:05:50,705 DEBUG [M:0;jenkins-hbase4:38491] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-15 19:05:50,705 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 19:05:50,709 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 19:05:50,709 INFO [M:0;jenkins-hbase4:38491] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-15 19:05:50,709 INFO [M:0;jenkins-hbase4:38491] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38491 2023-07-15 19:05:50,711 DEBUG [M:0;jenkins-hbase4:38491] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,38491,1689447924569 already deleted, retry=false 2023-07-15 19:05:50,934 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 19:05:50,934 INFO [M:0;jenkins-hbase4:38491] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,38491,1689447924569; zookeeper connection closed. 2023-07-15 19:05:50,934 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): master:38491-0x1016a6003100000, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 19:05:51,034 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:36163-0x1016a6003100008, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 19:05:51,034 INFO [RS:2;jenkins-hbase4:36163] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,36163,1689447931638; zookeeper connection closed. 2023-07-15 19:05:51,034 DEBUG [Listener at localhost/42557-EventThread] zookeeper.ZKWatcher(600): regionserver:36163-0x1016a6003100008, quorum=127.0.0.1:54942, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 19:05:51,035 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@64c9ed1f] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@64c9ed1f 2023-07-15 19:05:51,035 INFO [Listener at localhost/42557] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-15 19:05:51,035 WARN [Listener at localhost/42557] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-15 19:05:51,038 INFO [Listener at localhost/42557] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-15 19:05:51,142 WARN [BP-1710966795-172.31.14.131-1689447921569 heartbeating to localhost/127.0.0.1:45815] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-15 19:05:51,142 WARN [BP-1710966795-172.31.14.131-1689447921569 heartbeating to localhost/127.0.0.1:45815] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1710966795-172.31.14.131-1689447921569 (Datanode Uuid 1659b223-4fed-4869-94fe-7556636a6e4c) service to localhost/127.0.0.1:45815 2023-07-15 19:05:51,144 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/cluster_a9497dea-7554-f5e1-575e-fef856a8e7af/dfs/data/data3/current/BP-1710966795-172.31.14.131-1689447921569] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 19:05:51,144 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/cluster_a9497dea-7554-f5e1-575e-fef856a8e7af/dfs/data/data4/current/BP-1710966795-172.31.14.131-1689447921569] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 19:05:51,146 WARN [Listener at localhost/42557] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-15 19:05:51,148 INFO [Listener at localhost/42557] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-15 19:05:51,253 WARN [BP-1710966795-172.31.14.131-1689447921569 heartbeating to localhost/127.0.0.1:45815] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-15 19:05:51,253 WARN [BP-1710966795-172.31.14.131-1689447921569 heartbeating to localhost/127.0.0.1:45815] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1710966795-172.31.14.131-1689447921569 (Datanode Uuid c84b3dbf-c391-4f36-b298-3ef2e911d0d9) service to localhost/127.0.0.1:45815 2023-07-15 19:05:51,254 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/cluster_a9497dea-7554-f5e1-575e-fef856a8e7af/dfs/data/data1/current/BP-1710966795-172.31.14.131-1689447921569] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 19:05:51,254 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4220bb6b-35fe-ac7e-809a-558b64fdc5d4/cluster_a9497dea-7554-f5e1-575e-fef856a8e7af/dfs/data/data2/current/BP-1710966795-172.31.14.131-1689447921569] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 19:05:51,290 INFO [Listener at localhost/42557] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-15 19:05:51,403 INFO [Listener at localhost/42557] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-15 19:05:51,437 INFO [Listener at localhost/42557] hbase.HBaseTestingUtility(1293): Minicluster is down