2023-07-15 12:05:35,796 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-15 12:05:36,034 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af 2023-07-15 12:05:36,050 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-15 12:05:36,050 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/cluster_dc5c5ec5-5294-def2-fc28-e675bb2dd0fd, deleteOnExit=true 2023-07-15 12:05:36,050 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-15 12:05:36,051 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/test.cache.data in system properties and HBase conf 2023-07-15 12:05:36,051 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/hadoop.tmp.dir in system properties and HBase conf 2023-07-15 12:05:36,052 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/hadoop.log.dir in system properties and HBase conf 2023-07-15 12:05:36,052 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-15 12:05:36,052 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-15 12:05:36,053 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-15 12:05:36,166 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-15 12:05:36,559 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-15 12:05:36,562 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-15 12:05:36,563 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-15 12:05:36,563 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-15 12:05:36,563 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-15 12:05:36,564 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-15 12:05:36,564 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-15 12:05:36,564 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-15 12:05:36,564 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-15 12:05:36,565 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-15 12:05:36,565 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/nfs.dump.dir in system properties and HBase conf 2023-07-15 12:05:36,565 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/java.io.tmpdir in system properties and HBase conf 2023-07-15 12:05:36,565 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-15 12:05:36,566 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-15 12:05:36,566 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-15 12:05:37,049 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-15 12:05:37,053 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-15 12:05:37,328 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-15 12:05:37,488 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-15 12:05:37,508 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-15 12:05:37,543 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-15 12:05:37,574 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/java.io.tmpdir/Jetty_localhost_44563_hdfs____8f2zfg/webapp 2023-07-15 12:05:37,728 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44563 2023-07-15 12:05:37,739 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-15 12:05:37,739 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-15 12:05:38,159 WARN [Listener at localhost/43457] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-15 12:05:38,266 WARN [Listener at localhost/43457] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-15 12:05:38,284 WARN [Listener at localhost/43457] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-15 12:05:38,291 INFO [Listener at localhost/43457] log.Slf4jLog(67): jetty-6.1.26 2023-07-15 12:05:38,296 INFO [Listener at localhost/43457] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/java.io.tmpdir/Jetty_localhost_45373_datanode____.rd4i5n/webapp 2023-07-15 12:05:38,393 INFO [Listener at localhost/43457] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45373 2023-07-15 12:05:38,689 WARN [Listener at localhost/42017] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-15 12:05:38,699 WARN [Listener at localhost/42017] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-15 12:05:38,702 WARN [Listener at localhost/42017] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-15 12:05:38,705 INFO [Listener at localhost/42017] log.Slf4jLog(67): jetty-6.1.26 2023-07-15 12:05:38,711 INFO [Listener at localhost/42017] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/java.io.tmpdir/Jetty_localhost_42563_datanode____.eu8zol/webapp 2023-07-15 12:05:38,805 INFO [Listener at localhost/42017] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42563 2023-07-15 12:05:38,814 WARN [Listener at localhost/36687] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-15 12:05:39,146 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe1aec2ef18aeb36e: Processing first storage report for DS-f75ed93b-f0b5-4f50-8266-8678f5e04d08 from datanode 2482fd12-957d-4a8c-bdce-f56523fe2366 2023-07-15 12:05:39,148 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe1aec2ef18aeb36e: from storage DS-f75ed93b-f0b5-4f50-8266-8678f5e04d08 node DatanodeRegistration(127.0.0.1:34935, datanodeUuid=2482fd12-957d-4a8c-bdce-f56523fe2366, infoPort=35771, infoSecurePort=0, ipcPort=42017, storageInfo=lv=-57;cid=testClusterID;nsid=1394362140;c=1689422737126), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-15 12:05:39,148 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe5174b87a48e664c: Processing first storage report for DS-2f5970e4-45a6-4578-aaf0-34b1d5392eec from datanode 969e7c5c-c43d-4e03-bfdc-9128ceac1708 2023-07-15 12:05:39,148 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe5174b87a48e664c: from storage DS-2f5970e4-45a6-4578-aaf0-34b1d5392eec node DatanodeRegistration(127.0.0.1:40657, datanodeUuid=969e7c5c-c43d-4e03-bfdc-9128ceac1708, infoPort=41749, infoSecurePort=0, ipcPort=36687, storageInfo=lv=-57;cid=testClusterID;nsid=1394362140;c=1689422737126), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 12:05:39,148 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe1aec2ef18aeb36e: Processing first storage report for DS-ea201540-1f35-4b75-87ec-0f71c4d41ab4 from datanode 2482fd12-957d-4a8c-bdce-f56523fe2366 2023-07-15 12:05:39,149 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe1aec2ef18aeb36e: from storage DS-ea201540-1f35-4b75-87ec-0f71c4d41ab4 node DatanodeRegistration(127.0.0.1:34935, datanodeUuid=2482fd12-957d-4a8c-bdce-f56523fe2366, infoPort=35771, infoSecurePort=0, ipcPort=42017, storageInfo=lv=-57;cid=testClusterID;nsid=1394362140;c=1689422737126), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 12:05:39,149 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe5174b87a48e664c: Processing first storage report for DS-8b347300-e60c-43e5-ba97-7ac368b0651c from datanode 969e7c5c-c43d-4e03-bfdc-9128ceac1708 2023-07-15 12:05:39,149 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe5174b87a48e664c: from storage DS-8b347300-e60c-43e5-ba97-7ac368b0651c node DatanodeRegistration(127.0.0.1:40657, datanodeUuid=969e7c5c-c43d-4e03-bfdc-9128ceac1708, infoPort=41749, infoSecurePort=0, ipcPort=36687, storageInfo=lv=-57;cid=testClusterID;nsid=1394362140;c=1689422737126), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 12:05:39,203 DEBUG [Listener at localhost/36687] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af 2023-07-15 12:05:39,264 INFO [Listener at localhost/36687] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/cluster_dc5c5ec5-5294-def2-fc28-e675bb2dd0fd/zookeeper_0, clientPort=58992, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/cluster_dc5c5ec5-5294-def2-fc28-e675bb2dd0fd/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/cluster_dc5c5ec5-5294-def2-fc28-e675bb2dd0fd/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-15 12:05:39,280 INFO [Listener at localhost/36687] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58992 2023-07-15 12:05:39,291 INFO [Listener at localhost/36687] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 12:05:39,294 INFO [Listener at localhost/36687] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 12:05:39,948 INFO [Listener at localhost/36687] util.FSUtils(471): Created version file at hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c with version=8 2023-07-15 12:05:39,948 INFO [Listener at localhost/36687] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/hbase-staging 2023-07-15 12:05:40,259 INFO [Listener at localhost/36687] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-15 12:05:40,713 INFO [Listener at localhost/36687] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 12:05:40,745 INFO [Listener at localhost/36687] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 12:05:40,746 INFO [Listener at localhost/36687] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 12:05:40,746 INFO [Listener at localhost/36687] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 12:05:40,746 INFO [Listener at localhost/36687] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 12:05:40,747 INFO [Listener at localhost/36687] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 12:05:40,894 INFO [Listener at localhost/36687] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 12:05:40,965 DEBUG [Listener at localhost/36687] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-15 12:05:41,060 INFO [Listener at localhost/36687] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45429 2023-07-15 12:05:41,070 INFO [Listener at localhost/36687] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 12:05:41,072 INFO [Listener at localhost/36687] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 12:05:41,093 INFO [Listener at localhost/36687] zookeeper.RecoverableZooKeeper(93): Process identifier=master:45429 connecting to ZooKeeper ensemble=127.0.0.1:58992 2023-07-15 12:05:41,133 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:454290x0, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 12:05:41,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:45429-0x10168dfba350000 connected 2023-07-15 12:05:41,160 DEBUG [Listener at localhost/36687] zookeeper.ZKUtil(164): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 12:05:41,161 DEBUG [Listener at localhost/36687] zookeeper.ZKUtil(164): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 12:05:41,165 DEBUG [Listener at localhost/36687] zookeeper.ZKUtil(164): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 12:05:41,173 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45429 2023-07-15 12:05:41,173 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45429 2023-07-15 12:05:41,174 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45429 2023-07-15 12:05:41,174 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45429 2023-07-15 12:05:41,174 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45429 2023-07-15 12:05:41,180 INFO [Listener at localhost/36687] master.HMaster(444): hbase.rootdir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c, hbase.cluster.distributed=false 2023-07-15 12:05:41,247 INFO [Listener at localhost/36687] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 12:05:41,248 INFO [Listener at localhost/36687] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 12:05:41,248 INFO [Listener at localhost/36687] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 12:05:41,248 INFO [Listener at localhost/36687] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 12:05:41,248 INFO [Listener at localhost/36687] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 12:05:41,248 INFO [Listener at localhost/36687] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 12:05:41,253 INFO [Listener at localhost/36687] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 12:05:41,256 INFO [Listener at localhost/36687] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37725 2023-07-15 12:05:41,258 INFO [Listener at localhost/36687] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 12:05:41,264 DEBUG [Listener at localhost/36687] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 12:05:41,265 INFO [Listener at localhost/36687] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 12:05:41,267 INFO [Listener at localhost/36687] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 12:05:41,269 INFO [Listener at localhost/36687] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37725 connecting to ZooKeeper ensemble=127.0.0.1:58992 2023-07-15 12:05:41,272 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:377250x0, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 12:05:41,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:37725-0x10168dfba350001 connected 2023-07-15 12:05:41,273 DEBUG [Listener at localhost/36687] zookeeper.ZKUtil(164): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 12:05:41,275 DEBUG [Listener at localhost/36687] zookeeper.ZKUtil(164): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 12:05:41,276 DEBUG [Listener at localhost/36687] zookeeper.ZKUtil(164): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 12:05:41,277 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37725 2023-07-15 12:05:41,277 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37725 2023-07-15 12:05:41,278 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37725 2023-07-15 12:05:41,280 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37725 2023-07-15 12:05:41,280 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37725 2023-07-15 12:05:41,293 INFO [Listener at localhost/36687] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 12:05:41,293 INFO [Listener at localhost/36687] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 12:05:41,293 INFO [Listener at localhost/36687] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 12:05:41,294 INFO [Listener at localhost/36687] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 12:05:41,294 INFO [Listener at localhost/36687] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 12:05:41,294 INFO [Listener at localhost/36687] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 12:05:41,294 INFO [Listener at localhost/36687] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 12:05:41,297 INFO [Listener at localhost/36687] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46781 2023-07-15 12:05:41,298 INFO [Listener at localhost/36687] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 12:05:41,299 DEBUG [Listener at localhost/36687] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 12:05:41,300 INFO [Listener at localhost/36687] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 12:05:41,302 INFO [Listener at localhost/36687] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 12:05:41,303 INFO [Listener at localhost/36687] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46781 connecting to ZooKeeper ensemble=127.0.0.1:58992 2023-07-15 12:05:41,307 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:467810x0, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 12:05:41,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:46781-0x10168dfba350002 connected 2023-07-15 12:05:41,308 DEBUG [Listener at localhost/36687] zookeeper.ZKUtil(164): regionserver:46781-0x10168dfba350002, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 12:05:41,309 DEBUG [Listener at localhost/36687] zookeeper.ZKUtil(164): regionserver:46781-0x10168dfba350002, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 12:05:41,310 DEBUG [Listener at localhost/36687] zookeeper.ZKUtil(164): regionserver:46781-0x10168dfba350002, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 12:05:41,310 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46781 2023-07-15 12:05:41,311 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46781 2023-07-15 12:05:41,311 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46781 2023-07-15 12:05:41,312 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46781 2023-07-15 12:05:41,312 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46781 2023-07-15 12:05:41,313 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,45429,1689422740097 2023-07-15 12:05:41,324 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-15 12:05:41,325 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,45429,1689422740097 2023-07-15 12:05:41,343 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-15 12:05:41,343 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:46781-0x10168dfba350002, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-15 12:05:41,343 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-15 12:05:41,344 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 12:05:41,344 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 12:05:41,345 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,45429,1689422740097 from backup master directory 2023-07-15 12:05:41,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 12:05:41,349 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,45429,1689422740097 2023-07-15 12:05:41,349 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-15 12:05:41,349 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 12:05:41,350 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,45429,1689422740097 2023-07-15 12:05:41,352 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-15 12:05:41,354 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-15 12:05:41,439 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/hbase.id with ID: 594d7891-16a6-4fa8-a646-69890365aae2 2023-07-15 12:05:41,482 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 12:05:41,497 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 12:05:41,540 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0bee9c84 to 127.0.0.1:58992 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 12:05:41,574 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46782f3b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 12:05:41,596 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-15 12:05:41,599 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-15 12:05:41,618 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-15 12:05:41,618 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-15 12:05:41,620 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-15 12:05:41,624 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-15 12:05:41,625 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 12:05:41,663 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/MasterData/data/master/store-tmp 2023-07-15 12:05:41,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 12:05:41,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-15 12:05:41,696 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 12:05:41,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 12:05:41,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-15 12:05:41,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 12:05:41,696 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 12:05:41,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-15 12:05:41,698 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/MasterData/WALs/jenkins-hbase4.apache.org,45429,1689422740097 2023-07-15 12:05:41,718 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45429%2C1689422740097, suffix=, logDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/MasterData/WALs/jenkins-hbase4.apache.org,45429,1689422740097, archiveDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/MasterData/oldWALs, maxLogs=10 2023-07-15 12:05:41,772 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40657,DS-2f5970e4-45a6-4578-aaf0-34b1d5392eec,DISK] 2023-07-15 12:05:41,772 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34935,DS-f75ed93b-f0b5-4f50-8266-8678f5e04d08,DISK] 2023-07-15 12:05:41,779 DEBUG [RS-EventLoopGroup-4-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 12:05:41,849 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/MasterData/WALs/jenkins-hbase4.apache.org,45429,1689422740097/jenkins-hbase4.apache.org%2C45429%2C1689422740097.1689422741727 2023-07-15 12:05:41,850 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40657,DS-2f5970e4-45a6-4578-aaf0-34b1d5392eec,DISK], DatanodeInfoWithStorage[127.0.0.1:34935,DS-f75ed93b-f0b5-4f50-8266-8678f5e04d08,DISK]] 2023-07-15 12:05:41,850 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-15 12:05:41,851 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 12:05:41,854 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-15 12:05:41,855 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-15 12:05:41,910 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-15 12:05:41,917 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-15 12:05:41,940 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-15 12:05:41,951 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 12:05:41,957 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-15 12:05:41,959 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-15 12:05:41,973 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-15 12:05:41,977 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 12:05:41,978 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63181172, jitterRate=-0.05852717161178589}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 12:05:41,978 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-15 12:05:41,979 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-15 12:05:41,999 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-15 12:05:41,999 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-15 12:05:42,002 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-15 12:05:42,004 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-15 12:05:42,036 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-07-15 12:05:42,036 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-15 12:05:42,061 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-15 12:05:42,066 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-15 12:05:42,091 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-15 12:05:42,095 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-15 12:05:42,096 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-15 12:05:42,101 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-15 12:05:42,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-15 12:05:42,108 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 12:05:42,109 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-15 12:05:42,110 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-15 12:05:42,121 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-15 12:05:42,125 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-15 12:05:42,125 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:46781-0x10168dfba350002, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-15 12:05:42,125 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-15 12:05:42,126 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 12:05:42,126 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,45429,1689422740097, sessionid=0x10168dfba350000, setting cluster-up flag (Was=false) 2023-07-15 12:05:42,142 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 12:05:42,148 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-15 12:05:42,149 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,45429,1689422740097 2023-07-15 12:05:42,155 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 12:05:42,162 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-15 12:05:42,163 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,45429,1689422740097 2023-07-15 12:05:42,166 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/.hbase-snapshot/.tmp 2023-07-15 12:05:42,216 INFO [RS:1;jenkins-hbase4:46781] regionserver.HRegionServer(951): ClusterId : 594d7891-16a6-4fa8-a646-69890365aae2 2023-07-15 12:05:42,216 INFO [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(951): ClusterId : 594d7891-16a6-4fa8-a646-69890365aae2 2023-07-15 12:05:42,220 DEBUG [RS:1;jenkins-hbase4:46781] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 12:05:42,220 DEBUG [RS:0;jenkins-hbase4:37725] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 12:05:42,227 DEBUG [RS:0;jenkins-hbase4:37725] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 12:05:42,227 DEBUG [RS:0;jenkins-hbase4:37725] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 12:05:42,229 DEBUG [RS:1;jenkins-hbase4:46781] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 12:05:42,229 DEBUG [RS:1;jenkins-hbase4:46781] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 12:05:42,231 DEBUG [RS:0;jenkins-hbase4:37725] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 12:05:42,232 DEBUG [RS:0;jenkins-hbase4:37725] zookeeper.ReadOnlyZKClient(139): Connect 0x18ddedc4 to 127.0.0.1:58992 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 12:05:42,232 DEBUG [RS:1;jenkins-hbase4:46781] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 12:05:42,233 DEBUG [RS:1;jenkins-hbase4:46781] zookeeper.ReadOnlyZKClient(139): Connect 0x71673f46 to 127.0.0.1:58992 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 12:05:42,240 DEBUG [RS:0;jenkins-hbase4:37725] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@161a00af, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 12:05:42,241 DEBUG [RS:0;jenkins-hbase4:37725] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@122586b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 12:05:42,241 DEBUG [RS:1;jenkins-hbase4:46781] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7fa0cf1a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 12:05:42,241 DEBUG [RS:1;jenkins-hbase4:46781] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7b9f5436, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 12:05:42,267 DEBUG [RS:1;jenkins-hbase4:46781] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:46781 2023-07-15 12:05:42,267 DEBUG [RS:0;jenkins-hbase4:37725] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:37725 2023-07-15 12:05:42,272 INFO [RS:1;jenkins-hbase4:46781] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 12:05:42,272 INFO [RS:0;jenkins-hbase4:37725] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 12:05:42,272 INFO [RS:0;jenkins-hbase4:37725] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 12:05:42,272 INFO [RS:1;jenkins-hbase4:46781] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 12:05:42,272 DEBUG [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 12:05:42,272 DEBUG [RS:1;jenkins-hbase4:46781] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 12:05:42,275 INFO [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,45429,1689422740097 with isa=jenkins-hbase4.apache.org/172.31.14.131:37725, startcode=1689422741247 2023-07-15 12:05:42,275 INFO [RS:1;jenkins-hbase4:46781] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,45429,1689422740097 with isa=jenkins-hbase4.apache.org/172.31.14.131:46781, startcode=1689422741293 2023-07-15 12:05:42,284 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-15 12:05:42,294 DEBUG [RS:1;jenkins-hbase4:46781] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 12:05:42,294 DEBUG [RS:0;jenkins-hbase4:37725] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 12:05:42,295 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 12:05:42,295 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 12:05:42,296 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 12:05:42,296 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 12:05:42,296 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-15 12:05:42,296 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,296 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 12:05:42,296 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,297 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689422772297 2023-07-15 12:05:42,299 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-15 12:05:42,304 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-15 12:05:42,304 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-15 12:05:42,310 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-15 12:05:42,312 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-15 12:05:42,321 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-15 12:05:42,323 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-15 12:05:42,323 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-15 12:05:42,323 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-15 12:05:42,327 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:42,329 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-15 12:05:42,331 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-15 12:05:42,331 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-15 12:05:42,334 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-15 12:05:42,334 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-15 12:05:42,336 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689422742336,5,FailOnTimeoutGroup] 2023-07-15 12:05:42,337 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689422742336,5,FailOnTimeoutGroup] 2023-07-15 12:05:42,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:42,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-15 12:05:42,338 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:42,339 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:42,355 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-15 12:05:42,356 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-15 12:05:42,356 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c 2023-07-15 12:05:42,361 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43925, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 12:05:42,361 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36091, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 12:05:42,378 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 12:05:42,379 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45429] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,46781,1689422741293 2023-07-15 12:05:42,383 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 12:05:42,386 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/info 2023-07-15 12:05:42,386 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 12:05:42,388 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 12:05:42,388 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 12:05:42,391 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45429] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:42,392 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/rep_barrier 2023-07-15 12:05:42,393 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 12:05:42,394 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 12:05:42,394 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 12:05:42,396 DEBUG [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c 2023-07-15 12:05:42,396 DEBUG [RS:1;jenkins-hbase4:46781] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c 2023-07-15 12:05:42,397 DEBUG [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:43457 2023-07-15 12:05:42,397 DEBUG [RS:1;jenkins-hbase4:46781] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:43457 2023-07-15 12:05:42,397 DEBUG [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-15 12:05:42,397 DEBUG [RS:1;jenkins-hbase4:46781] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-15 12:05:42,398 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/table 2023-07-15 12:05:42,398 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 12:05:42,399 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 12:05:42,401 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740 2023-07-15 12:05:42,403 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740 2023-07-15 12:05:42,404 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 12:05:42,409 DEBUG [RS:0;jenkins-hbase4:37725] zookeeper.ZKUtil(162): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:42,409 WARN [RS:0;jenkins-hbase4:37725] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 12:05:42,410 INFO [RS:0;jenkins-hbase4:37725] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 12:05:42,410 DEBUG [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(1948): logDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:42,410 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,46781,1689422741293] 2023-07-15 12:05:42,411 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,37725,1689422741247] 2023-07-15 12:05:42,411 DEBUG [RS:1;jenkins-hbase4:46781] zookeeper.ZKUtil(162): regionserver:46781-0x10168dfba350002, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46781,1689422741293 2023-07-15 12:05:42,411 WARN [RS:1;jenkins-hbase4:46781] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 12:05:42,411 INFO [RS:1;jenkins-hbase4:46781] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 12:05:42,412 DEBUG [RS:1;jenkins-hbase4:46781] regionserver.HRegionServer(1948): logDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293 2023-07-15 12:05:42,417 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 12:05:42,419 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 12:05:42,430 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 12:05:42,432 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72668493, jitterRate=0.08284492790699005}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 12:05:42,432 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 12:05:42,432 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 12:05:42,432 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 12:05:42,432 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 12:05:42,433 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 12:05:42,433 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 12:05:42,434 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 12:05:42,434 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 12:05:42,435 DEBUG [RS:1;jenkins-hbase4:46781] zookeeper.ZKUtil(162): regionserver:46781-0x10168dfba350002, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46781,1689422741293 2023-07-15 12:05:42,435 DEBUG [RS:0;jenkins-hbase4:37725] zookeeper.ZKUtil(162): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46781,1689422741293 2023-07-15 12:05:42,435 DEBUG [RS:1;jenkins-hbase4:46781] zookeeper.ZKUtil(162): regionserver:46781-0x10168dfba350002, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:42,436 DEBUG [RS:0;jenkins-hbase4:37725] zookeeper.ZKUtil(162): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:42,440 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-15 12:05:42,440 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-15 12:05:42,448 DEBUG [RS:1;jenkins-hbase4:46781] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 12:05:42,448 DEBUG [RS:0;jenkins-hbase4:37725] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 12:05:42,449 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-15 12:05:42,458 INFO [RS:1;jenkins-hbase4:46781] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 12:05:42,458 INFO [RS:0;jenkins-hbase4:37725] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 12:05:42,462 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-15 12:05:42,465 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-15 12:05:42,480 INFO [RS:1;jenkins-hbase4:46781] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 12:05:42,480 INFO [RS:0;jenkins-hbase4:37725] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 12:05:42,484 INFO [RS:1;jenkins-hbase4:46781] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 12:05:42,484 INFO [RS:0;jenkins-hbase4:37725] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 12:05:42,485 INFO [RS:1;jenkins-hbase4:46781] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:42,485 INFO [RS:0;jenkins-hbase4:37725] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:42,486 INFO [RS:1;jenkins-hbase4:46781] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 12:05:42,486 INFO [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 12:05:42,494 INFO [RS:0;jenkins-hbase4:37725] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:42,494 INFO [RS:1;jenkins-hbase4:46781] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:42,494 DEBUG [RS:0;jenkins-hbase4:37725] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,494 DEBUG [RS:1;jenkins-hbase4:46781] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,495 DEBUG [RS:0;jenkins-hbase4:37725] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,495 DEBUG [RS:1;jenkins-hbase4:46781] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,495 DEBUG [RS:0;jenkins-hbase4:37725] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,495 DEBUG [RS:1;jenkins-hbase4:46781] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,495 DEBUG [RS:0;jenkins-hbase4:37725] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,495 DEBUG [RS:1;jenkins-hbase4:46781] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,495 DEBUG [RS:0;jenkins-hbase4:37725] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,495 DEBUG [RS:1;jenkins-hbase4:46781] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,495 DEBUG [RS:0;jenkins-hbase4:37725] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 12:05:42,496 DEBUG [RS:1;jenkins-hbase4:46781] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 12:05:42,496 DEBUG [RS:0;jenkins-hbase4:37725] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,496 DEBUG [RS:1;jenkins-hbase4:46781] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,496 DEBUG [RS:0;jenkins-hbase4:37725] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,496 DEBUG [RS:1;jenkins-hbase4:46781] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,496 DEBUG [RS:0;jenkins-hbase4:37725] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,496 DEBUG [RS:1;jenkins-hbase4:46781] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,496 DEBUG [RS:0;jenkins-hbase4:37725] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,496 DEBUG [RS:1;jenkins-hbase4:46781] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:42,499 INFO [RS:1;jenkins-hbase4:46781] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:42,499 INFO [RS:1;jenkins-hbase4:46781] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:42,499 INFO [RS:0;jenkins-hbase4:37725] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:42,499 INFO [RS:1;jenkins-hbase4:46781] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:42,500 INFO [RS:0;jenkins-hbase4:37725] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:42,500 INFO [RS:0;jenkins-hbase4:37725] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:42,515 INFO [RS:1;jenkins-hbase4:46781] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 12:05:42,515 INFO [RS:0;jenkins-hbase4:37725] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 12:05:42,518 INFO [RS:0;jenkins-hbase4:37725] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37725,1689422741247-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:42,518 INFO [RS:1;jenkins-hbase4:46781] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46781,1689422741293-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:42,536 INFO [RS:1;jenkins-hbase4:46781] regionserver.Replication(203): jenkins-hbase4.apache.org,46781,1689422741293 started 2023-07-15 12:05:42,536 INFO [RS:0;jenkins-hbase4:37725] regionserver.Replication(203): jenkins-hbase4.apache.org,37725,1689422741247 started 2023-07-15 12:05:42,536 INFO [RS:1;jenkins-hbase4:46781] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,46781,1689422741293, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:46781, sessionid=0x10168dfba350002 2023-07-15 12:05:42,536 INFO [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,37725,1689422741247, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:37725, sessionid=0x10168dfba350001 2023-07-15 12:05:42,537 DEBUG [RS:1;jenkins-hbase4:46781] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 12:05:42,537 DEBUG [RS:0;jenkins-hbase4:37725] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 12:05:42,537 DEBUG [RS:1;jenkins-hbase4:46781] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,46781,1689422741293 2023-07-15 12:05:42,537 DEBUG [RS:0;jenkins-hbase4:37725] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:42,537 DEBUG [RS:1;jenkins-hbase4:46781] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46781,1689422741293' 2023-07-15 12:05:42,538 DEBUG [RS:0;jenkins-hbase4:37725] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37725,1689422741247' 2023-07-15 12:05:42,538 DEBUG [RS:0;jenkins-hbase4:37725] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 12:05:42,538 DEBUG [RS:1;jenkins-hbase4:46781] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 12:05:42,539 DEBUG [RS:1;jenkins-hbase4:46781] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 12:05:42,539 DEBUG [RS:0;jenkins-hbase4:37725] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 12:05:42,540 DEBUG [RS:1;jenkins-hbase4:46781] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 12:05:42,540 DEBUG [RS:1;jenkins-hbase4:46781] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 12:05:42,540 DEBUG [RS:0;jenkins-hbase4:37725] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 12:05:42,540 DEBUG [RS:1;jenkins-hbase4:46781] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,46781,1689422741293 2023-07-15 12:05:42,540 DEBUG [RS:1;jenkins-hbase4:46781] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46781,1689422741293' 2023-07-15 12:05:42,540 DEBUG [RS:1;jenkins-hbase4:46781] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 12:05:42,540 DEBUG [RS:0;jenkins-hbase4:37725] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 12:05:42,540 DEBUG [RS:0;jenkins-hbase4:37725] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:42,541 DEBUG [RS:0;jenkins-hbase4:37725] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37725,1689422741247' 2023-07-15 12:05:42,541 DEBUG [RS:0;jenkins-hbase4:37725] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 12:05:42,541 DEBUG [RS:1;jenkins-hbase4:46781] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 12:05:42,541 DEBUG [RS:0;jenkins-hbase4:37725] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 12:05:42,542 DEBUG [RS:1;jenkins-hbase4:46781] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 12:05:42,542 INFO [RS:1;jenkins-hbase4:46781] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 12:05:42,542 INFO [RS:1;jenkins-hbase4:46781] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 12:05:42,542 DEBUG [RS:0;jenkins-hbase4:37725] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 12:05:42,542 INFO [RS:0;jenkins-hbase4:37725] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 12:05:42,543 INFO [RS:0;jenkins-hbase4:37725] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 12:05:42,617 DEBUG [jenkins-hbase4:45429] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-15 12:05:42,622 DEBUG [jenkins-hbase4:45429] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 12:05:42,629 DEBUG [jenkins-hbase4:45429] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 12:05:42,629 DEBUG [jenkins-hbase4:45429] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 12:05:42,629 DEBUG [jenkins-hbase4:45429] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 12:05:42,632 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,37725,1689422741247, state=OPENING 2023-07-15 12:05:42,639 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-15 12:05:42,641 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 12:05:42,641 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 12:05:42,645 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,37725,1689422741247}] 2023-07-15 12:05:42,655 INFO [RS:1;jenkins-hbase4:46781] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46781%2C1689422741293, suffix=, logDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293, archiveDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/oldWALs, maxLogs=32 2023-07-15 12:05:42,655 INFO [RS:0;jenkins-hbase4:37725] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37725%2C1689422741247, suffix=, logDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247, archiveDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/oldWALs, maxLogs=32 2023-07-15 12:05:42,681 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34935,DS-f75ed93b-f0b5-4f50-8266-8678f5e04d08,DISK] 2023-07-15 12:05:42,681 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40657,DS-2f5970e4-45a6-4578-aaf0-34b1d5392eec,DISK] 2023-07-15 12:05:42,681 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40657,DS-2f5970e4-45a6-4578-aaf0-34b1d5392eec,DISK] 2023-07-15 12:05:42,682 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34935,DS-f75ed93b-f0b5-4f50-8266-8678f5e04d08,DISK] 2023-07-15 12:05:42,691 INFO [RS:0;jenkins-hbase4:37725] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247/jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660 2023-07-15 12:05:42,691 INFO [RS:1;jenkins-hbase4:46781] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293/jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660 2023-07-15 12:05:42,692 DEBUG [RS:0;jenkins-hbase4:37725] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34935,DS-f75ed93b-f0b5-4f50-8266-8678f5e04d08,DISK], DatanodeInfoWithStorage[127.0.0.1:40657,DS-2f5970e4-45a6-4578-aaf0-34b1d5392eec,DISK]] 2023-07-15 12:05:42,692 DEBUG [RS:1;jenkins-hbase4:46781] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40657,DS-2f5970e4-45a6-4578-aaf0-34b1d5392eec,DISK], DatanodeInfoWithStorage[127.0.0.1:34935,DS-f75ed93b-f0b5-4f50-8266-8678f5e04d08,DISK]] 2023-07-15 12:05:42,834 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:42,836 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 12:05:42,840 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57918, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 12:05:42,851 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-15 12:05:42,851 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 12:05:42,854 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta, suffix=.meta, logDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247, archiveDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/oldWALs, maxLogs=32 2023-07-15 12:05:42,872 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40657,DS-2f5970e4-45a6-4578-aaf0-34b1d5392eec,DISK] 2023-07-15 12:05:42,872 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34935,DS-f75ed93b-f0b5-4f50-8266-8678f5e04d08,DISK] 2023-07-15 12:05:42,879 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247/jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta 2023-07-15 12:05:42,880 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40657,DS-2f5970e4-45a6-4578-aaf0-34b1d5392eec,DISK], DatanodeInfoWithStorage[127.0.0.1:34935,DS-f75ed93b-f0b5-4f50-8266-8678f5e04d08,DISK]] 2023-07-15 12:05:42,880 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-15 12:05:42,882 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 12:05:42,904 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-15 12:05:42,923 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-15 12:05:42,928 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-15 12:05:42,928 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 12:05:42,928 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-15 12:05:42,928 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-15 12:05:42,931 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 12:05:42,933 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/info 2023-07-15 12:05:42,933 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/info 2023-07-15 12:05:42,933 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 12:05:42,934 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 12:05:42,935 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 12:05:42,936 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/rep_barrier 2023-07-15 12:05:42,936 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/rep_barrier 2023-07-15 12:05:42,937 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 12:05:42,937 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 12:05:42,938 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 12:05:42,939 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/table 2023-07-15 12:05:42,939 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/table 2023-07-15 12:05:42,939 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 12:05:42,940 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 12:05:42,942 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740 2023-07-15 12:05:42,944 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740 2023-07-15 12:05:42,948 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 12:05:42,950 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 12:05:42,951 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75321159, jitterRate=0.12237273156642914}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 12:05:42,951 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 12:05:42,962 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689422742826 2023-07-15 12:05:42,978 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-15 12:05:42,979 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-15 12:05:42,980 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,37725,1689422741247, state=OPEN 2023-07-15 12:05:42,983 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 12:05:42,983 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 12:05:42,987 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-15 12:05:42,987 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,37725,1689422741247 in 338 msec 2023-07-15 12:05:42,991 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-15 12:05:42,992 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 539 msec 2023-07-15 12:05:42,996 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 787 msec 2023-07-15 12:05:42,997 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689422742997, completionTime=-1 2023-07-15 12:05:42,997 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-15 12:05:42,997 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-15 12:05:43,064 DEBUG [hconnection-0x1b885658-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 12:05:43,067 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57926, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 12:05:43,085 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-15 12:05:43,085 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689422803085 2023-07-15 12:05:43,086 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689422863086 2023-07-15 12:05:43,086 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 88 msec 2023-07-15 12:05:43,109 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45429,1689422740097-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:43,109 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45429,1689422740097-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:43,109 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45429,1689422740097-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:43,111 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:45429, period=300000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:43,111 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:43,117 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-15 12:05:43,126 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-15 12:05:43,128 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-15 12:05:43,138 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-15 12:05:43,141 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-15 12:05:43,143 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-15 12:05:43,166 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/.tmp/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:05:43,168 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/.tmp/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4 empty. 2023-07-15 12:05:43,169 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/.tmp/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:05:43,169 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-15 12:05:43,196 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-15 12:05:43,198 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => b84ff22b66b7d5e955328627d26038b4, NAME => 'hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/.tmp 2023-07-15 12:05:43,214 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 12:05:43,214 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing b84ff22b66b7d5e955328627d26038b4, disabling compactions & flushes 2023-07-15 12:05:43,214 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:05:43,214 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:05:43,214 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. after waiting 0 ms 2023-07-15 12:05:43,214 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:05:43,214 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:05:43,214 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for b84ff22b66b7d5e955328627d26038b4: 2023-07-15 12:05:43,218 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-15 12:05:43,235 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689422743221"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689422743221"}]},"ts":"1689422743221"} 2023-07-15 12:05:43,265 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-15 12:05:43,267 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-15 12:05:43,271 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689422743267"}]},"ts":"1689422743267"} 2023-07-15 12:05:43,276 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-15 12:05:43,279 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 12:05:43,281 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 12:05:43,281 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 12:05:43,281 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 12:05:43,283 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=b84ff22b66b7d5e955328627d26038b4, ASSIGN}] 2023-07-15 12:05:43,286 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=b84ff22b66b7d5e955328627d26038b4, ASSIGN 2023-07-15 12:05:43,288 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=b84ff22b66b7d5e955328627d26038b4, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37725,1689422741247; forceNewPlan=false, retain=false 2023-07-15 12:05:43,441 INFO [jenkins-hbase4:45429] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-15 12:05:43,442 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=b84ff22b66b7d5e955328627d26038b4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:43,442 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689422743442"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689422743442"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689422743442"}]},"ts":"1689422743442"} 2023-07-15 12:05:43,448 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure b84ff22b66b7d5e955328627d26038b4, server=jenkins-hbase4.apache.org,37725,1689422741247}] 2023-07-15 12:05:43,609 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:05:43,610 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b84ff22b66b7d5e955328627d26038b4, NAME => 'hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4.', STARTKEY => '', ENDKEY => ''} 2023-07-15 12:05:43,611 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:05:43,611 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 12:05:43,611 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:05:43,611 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:05:43,613 INFO [StoreOpener-b84ff22b66b7d5e955328627d26038b4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:05:43,615 DEBUG [StoreOpener-b84ff22b66b7d5e955328627d26038b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/info 2023-07-15 12:05:43,615 DEBUG [StoreOpener-b84ff22b66b7d5e955328627d26038b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/info 2023-07-15 12:05:43,616 INFO [StoreOpener-b84ff22b66b7d5e955328627d26038b4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b84ff22b66b7d5e955328627d26038b4 columnFamilyName info 2023-07-15 12:05:43,617 INFO [StoreOpener-b84ff22b66b7d5e955328627d26038b4-1] regionserver.HStore(310): Store=b84ff22b66b7d5e955328627d26038b4/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 12:05:43,619 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:05:43,619 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:05:43,624 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:05:43,629 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 12:05:43,630 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b84ff22b66b7d5e955328627d26038b4; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61484884, jitterRate=-0.08380383253097534}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 12:05:43,630 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b84ff22b66b7d5e955328627d26038b4: 2023-07-15 12:05:43,632 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4., pid=6, masterSystemTime=1689422743602 2023-07-15 12:05:43,638 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:05:43,638 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:05:43,639 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=b84ff22b66b7d5e955328627d26038b4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:43,639 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689422743638"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689422743638"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689422743638"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689422743638"}]},"ts":"1689422743638"} 2023-07-15 12:05:43,646 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-15 12:05:43,646 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure b84ff22b66b7d5e955328627d26038b4, server=jenkins-hbase4.apache.org,37725,1689422741247 in 194 msec 2023-07-15 12:05:43,650 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-15 12:05:43,650 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=b84ff22b66b7d5e955328627d26038b4, ASSIGN in 363 msec 2023-07-15 12:05:43,651 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-15 12:05:43,651 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689422743651"}]},"ts":"1689422743651"} 2023-07-15 12:05:43,654 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-15 12:05:43,659 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-15 12:05:43,662 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 530 msec 2023-07-15 12:05:43,741 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-15 12:05:43,743 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-15 12:05:43,743 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 12:05:43,781 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-15 12:05:43,800 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-15 12:05:43,806 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-07-15 12:05:43,815 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-15 12:05:43,827 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-15 12:05:43,832 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-07-15 12:05:43,841 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-15 12:05:43,844 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-15 12:05:43,844 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.494sec 2023-07-15 12:05:43,847 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-15 12:05:43,849 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-15 12:05:43,849 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-15 12:05:43,850 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45429,1689422740097-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-15 12:05:43,851 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45429,1689422740097-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-15 12:05:43,860 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-15 12:05:43,922 DEBUG [Listener at localhost/36687] zookeeper.ReadOnlyZKClient(139): Connect 0x4789b34a to 127.0.0.1:58992 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 12:05:43,927 DEBUG [Listener at localhost/36687] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44a9743, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 12:05:43,940 DEBUG [hconnection-0x4e56a287-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 12:05:43,953 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57942, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 12:05:43,964 INFO [Listener at localhost/36687] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,45429,1689422740097 2023-07-15 12:05:43,976 DEBUG [Listener at localhost/36687] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-15 12:05:43,979 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54836, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-15 12:05:43,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45429] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-15 12:05:43,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45429] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-15 12:05:43,996 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-15 12:05:43,998 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-15 12:05:44,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45429] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-15 12:05:44,001 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/.tmp/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:44,002 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/.tmp/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd empty. 2023-07-15 12:05:44,004 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/.tmp/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:44,004 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-15 12:05:44,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45429] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-15 12:05:44,024 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-15 12:05:44,026 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => e58b75ac0a2024c3a9f998ec952af1bd, NAME => 't,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/.tmp 2023-07-15 12:05:44,042 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 12:05:44,043 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing e58b75ac0a2024c3a9f998ec952af1bd, disabling compactions & flushes 2023-07-15 12:05:44,043 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:44,043 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:44,043 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. after waiting 0 ms 2023-07-15 12:05:44,043 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:44,043 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:44,043 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for e58b75ac0a2024c3a9f998ec952af1bd: 2023-07-15 12:05:44,047 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-15 12:05:44,049 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689422744049"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689422744049"}]},"ts":"1689422744049"} 2023-07-15 12:05:44,051 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-15 12:05:44,053 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-15 12:05:44,053 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689422744053"}]},"ts":"1689422744053"} 2023-07-15 12:05:44,055 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-15 12:05:44,060 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 12:05:44,061 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 12:05:44,062 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 12:05:44,062 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 12:05:44,062 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=e58b75ac0a2024c3a9f998ec952af1bd, ASSIGN}] 2023-07-15 12:05:44,064 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=e58b75ac0a2024c3a9f998ec952af1bd, ASSIGN 2023-07-15 12:05:44,065 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=e58b75ac0a2024c3a9f998ec952af1bd, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46781,1689422741293; forceNewPlan=false, retain=false 2023-07-15 12:05:44,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45429] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-15 12:05:44,215 INFO [jenkins-hbase4:45429] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-15 12:05:44,216 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=e58b75ac0a2024c3a9f998ec952af1bd, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46781,1689422741293 2023-07-15 12:05:44,217 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689422744216"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689422744216"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689422744216"}]},"ts":"1689422744216"} 2023-07-15 12:05:44,220 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure e58b75ac0a2024c3a9f998ec952af1bd, server=jenkins-hbase4.apache.org,46781,1689422741293}] 2023-07-15 12:05:44,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45429] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-15 12:05:44,374 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,46781,1689422741293 2023-07-15 12:05:44,374 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 12:05:44,377 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55294, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 12:05:44,382 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:44,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e58b75ac0a2024c3a9f998ec952af1bd, NAME => 't,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.', STARTKEY => '', ENDKEY => ''} 2023-07-15 12:05:44,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:44,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 12:05:44,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:44,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:44,386 INFO [StoreOpener-e58b75ac0a2024c3a9f998ec952af1bd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:44,388 DEBUG [StoreOpener-e58b75ac0a2024c3a9f998ec952af1bd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/f 2023-07-15 12:05:44,388 DEBUG [StoreOpener-e58b75ac0a2024c3a9f998ec952af1bd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/f 2023-07-15 12:05:44,389 INFO [StoreOpener-e58b75ac0a2024c3a9f998ec952af1bd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e58b75ac0a2024c3a9f998ec952af1bd columnFamilyName f 2023-07-15 12:05:44,390 INFO [StoreOpener-e58b75ac0a2024c3a9f998ec952af1bd-1] regionserver.HStore(310): Store=e58b75ac0a2024c3a9f998ec952af1bd/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 12:05:44,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:44,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:44,396 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:44,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 12:05:44,401 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e58b75ac0a2024c3a9f998ec952af1bd; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59436854, jitterRate=-0.11432185769081116}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 12:05:44,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e58b75ac0a2024c3a9f998ec952af1bd: 2023-07-15 12:05:44,402 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd., pid=11, masterSystemTime=1689422744374 2023-07-15 12:05:44,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:44,407 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:44,408 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=e58b75ac0a2024c3a9f998ec952af1bd, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46781,1689422741293 2023-07-15 12:05:44,408 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689422744408"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689422744408"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689422744408"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689422744408"}]},"ts":"1689422744408"} 2023-07-15 12:05:44,414 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-15 12:05:44,414 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure e58b75ac0a2024c3a9f998ec952af1bd, server=jenkins-hbase4.apache.org,46781,1689422741293 in 191 msec 2023-07-15 12:05:44,417 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-15 12:05:44,418 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=e58b75ac0a2024c3a9f998ec952af1bd, ASSIGN in 352 msec 2023-07-15 12:05:44,419 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-15 12:05:44,419 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689422744419"}]},"ts":"1689422744419"} 2023-07-15 12:05:44,421 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-15 12:05:44,424 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-15 12:05:44,427 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 432 msec 2023-07-15 12:05:44,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45429] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-15 12:05:44,621 INFO [Listener at localhost/36687] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-15 12:05:44,622 DEBUG [Listener at localhost/36687] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-15 12:05:44,623 INFO [Listener at localhost/36687] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 12:05:44,627 INFO [Listener at localhost/36687] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-15 12:05:44,628 INFO [Listener at localhost/36687] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 12:05:44,628 INFO [Listener at localhost/36687] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-15 12:05:44,631 DEBUG [Listener at localhost/36687] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 12:05:44,634 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55306, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 12:05:44,678 INFO [Listener at localhost/36687] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=295, OpenFileDescriptor=585, MaxFileDescriptor=60000, SystemLoadAverage=150, ProcessCount=170, AvailableMemoryMB=6373 2023-07-15 12:05:44,680 INFO [Listener at localhost/36687] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-15 12:05:44,684 DEBUG [Listener at localhost/36687] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-15 12:05:44,684 DEBUG [Listener at localhost/36687] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:44,685 INFO [Listener at localhost/36687] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-15 12:05:44,685 INFO [Listener at localhost/36687] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-15 12:05:44,688 INFO [Listener at localhost/36687] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689422743651/Put/vlen=2/seqid=0} 2023-07-15 12:05:44,688 INFO [Listener at localhost/36687] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-15 12:05:44,688 INFO [Listener at localhost/36687] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4./info:regioninfo/1689422743638/Put/vlen=41/seqid=0, hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4./info:seqnumDuringOpen/1689422743638/Put/vlen=8/seqid=0, hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4./info:server/1689422743638/Put/vlen=31/seqid=0, hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4./info:serverstartcode/1689422743638/Put/vlen=8/seqid=0, hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4./info:sn/1689422743442/Put/vlen=45/seqid=0, hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4./info:state/1689422743638/Put/vlen=4/seqid=0} 2023-07-15 12:05:44,689 INFO [Listener at localhost/36687] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => b84ff22b66b7d5e955328627d26038b4, NAME => 'hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4.', STARTKEY => '', ENDKEY => ''} 2023-07-15 12:05:44,689 INFO [Listener at localhost/36687] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689422744419/Put/vlen=2/seqid=0} 2023-07-15 12:05:44,689 INFO [Listener at localhost/36687] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-15 12:05:44,689 INFO [Listener at localhost/36687] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd./info:regioninfo/1689422744408/Put/vlen=35/seqid=0, t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd./info:seqnumDuringOpen/1689422744408/Put/vlen=8/seqid=0, t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd./info:server/1689422744408/Put/vlen=31/seqid=0, t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd./info:serverstartcode/1689422744408/Put/vlen=8/seqid=0, t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd./info:sn/1689422744216/Put/vlen=45/seqid=0, t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd./info:state/1689422744408/Put/vlen=4/seqid=0} 2023-07-15 12:05:44,689 INFO [Listener at localhost/36687] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => e58b75ac0a2024c3a9f998ec952af1bd, NAME => 't,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.', STARTKEY => '', ENDKEY => ''} 2023-07-15 12:05:44,690 DEBUG [Listener at localhost/36687] zookeeper.ReadOnlyZKClient(139): Connect 0x1e8cbdf7 to 127.0.0.1:58992 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 12:05:44,695 DEBUG [Listener at localhost/36687] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@420100ae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 12:05:44,695 INFO [Listener at localhost/36687] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-15 12:05:44,695 INFO [Listener at localhost/36687] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-15 12:05:44,697 DEBUG [hconnection-0x31a3e955-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 12:05:44,700 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57950, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 12:05:44,704 DEBUG [hconnection-0x31a3e955-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 12:05:44,706 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55308, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 12:05:44,708 INFO [Listener at localhost/36687] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-15 12:05:44,711 ERROR [Listener at localhost/36687] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,46781,1689422741293: die! ***** 2023-07-15 12:05:44,712 ERROR [Listener at localhost/36687] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-15 12:05:44,715 DEBUG [Listener at localhost/36687] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-15 12:05:44,718 DEBUG [Listener at localhost/36687] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-15 12:05:44,718 DEBUG [Listener at localhost/36687] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-15 12:05:44,719 DEBUG [Listener at localhost/36687] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-15 12:05:44,719 INFO [Listener at localhost/36687] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 877133824, "init": 513802240, "max": 2051014656, "used": 397709504 }, "NonHeapMemoryUsage": { "committed": 113074176, "init": 2555904, "max": -1, "used": 111097080 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-15 12:05:44,723 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45429] master.MasterRpcServices(612): jenkins-hbase4.apache.org,46781,1689422741293 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,46781,1689422741293: die! ***** 2023-07-15 12:05:44,725 INFO [Listener at localhost/36687] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,46781,1689422741293' ***** 2023-07-15 12:05:44,726 INFO [Listener at localhost/36687] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-15 12:05:44,726 INFO [RS:1;jenkins-hbase4:46781] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 12:05:44,726 INFO [RS:1;jenkins-hbase4:46781] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-15 12:05:44,726 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 12:05:44,726 INFO [RS:1;jenkins-hbase4:46781] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-15 12:05:44,727 INFO [RS:1;jenkins-hbase4:46781] regionserver.HRegionServer(3305): Received CLOSE for e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:44,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46781] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner -5647177971790774271 2023-07-15 12:05:44,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46781] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,46781,1689422741293 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 12:05:44,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e58b75ac0a2024c3a9f998ec952af1bd, disabling compactions & flushes 2023-07-15 12:05:44,727 INFO [RS:1;jenkins-hbase4:46781] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,46781,1689422741293 2023-07-15 12:05:44,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46781] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 172.31.14.131:55308 deadline: 1689422804727, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,46781,1689422741293 aborting 2023-07-15 12:05:44,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:44,728 DEBUG [RS:1;jenkins-hbase4:46781] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71673f46 to 127.0.0.1:58992 2023-07-15 12:05:44,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:44,728 DEBUG [RS:1;jenkins-hbase4:46781] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 12:05:44,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. after waiting 0 ms 2023-07-15 12:05:44,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:44,729 INFO [RS:1;jenkins-hbase4:46781] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-15 12:05:44,729 DEBUG [RS:1;jenkins-hbase4:46781] regionserver.HRegionServer(1478): Online Regions={e58b75ac0a2024c3a9f998ec952af1bd=t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.} 2023-07-15 12:05:44,729 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:44,730 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:44,730 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e58b75ac0a2024c3a9f998ec952af1bd: 2023-07-15 12:05:44,730 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:44,730 DEBUG [RS:1;jenkins-hbase4:46781] regionserver.HRegionServer(1504): Waiting on 2023-07-15 12:05:44,746 DEBUG [Listener at localhost/36687] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,46781,1689422741293 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,46781,1689422741293 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-15 12:05:44,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46781] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,46781,1689422741293 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 12:05:44,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46781] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:55308 deadline: 1689422804749, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,46781,1689422741293 aborting 2023-07-15 12:05:44,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46781] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,46781,1689422741293 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 12:05:44,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46781] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:55308 deadline: 1689422804858, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,46781,1689422741293 aborting 2023-07-15 12:05:44,931 INFO [RS:1;jenkins-hbase4:46781] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,46781,1689422741293; all regions closed. 2023-07-15 12:05:44,941 DEBUG [RS:1;jenkins-hbase4:46781] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 12:05:44,942 INFO [RS:1;jenkins-hbase4:46781] regionserver.LeaseManager(133): Closed leases 2023-07-15 12:05:44,942 INFO [RS:1;jenkins-hbase4:46781] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-15 12:05:44,942 INFO [RS:1;jenkins-hbase4:46781] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-15 12:05:44,942 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 12:05:44,942 INFO [RS:1;jenkins-hbase4:46781] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-15 12:05:44,942 INFO [RS:1;jenkins-hbase4:46781] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-15 12:05:44,943 INFO [RS:1;jenkins-hbase4:46781] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46781 2023-07-15 12:05:44,983 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:46781-0x10168dfba350002, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46781,1689422741293 2023-07-15 12:05:44,983 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46781,1689422741293 2023-07-15 12:05:44,983 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 12:05:44,983 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 12:05:44,983 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:46781-0x10168dfba350002, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 12:05:44,985 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,46781,1689422741293] 2023-07-15 12:05:44,985 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,46781,1689422741293; numProcessing=1 2023-07-15 12:05:44,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:44,986 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,46781,1689422741293 znode expired, triggering replicatorRemoved event 2023-07-15 12:05:45,065 WARN [RPCClient-NioEventLoopGroup-5-5] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:46781 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46781 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 12:05:45,066 DEBUG [RPCClient-NioEventLoopGroup-5-5] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:46781 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46781 2023-07-15 12:05:45,085 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:46781-0x10168dfba350002, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 12:05:45,086 INFO [RS:1;jenkins-hbase4:46781] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,46781,1689422741293; zookeeper connection closed. 2023-07-15 12:05:45,086 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:46781-0x10168dfba350002, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 12:05:45,086 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2c874e40] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2c874e40 2023-07-15 12:05:45,087 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,46781,1689422741293 already deleted, retry=false 2023-07-15 12:05:45,087 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,46781,1689422741293 on jenkins-hbase4.apache.org,45429,1689422740097 2023-07-15 12:05:45,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:45,095 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,46781,1689422741293, splitWal=true, meta=false 2023-07-15 12:05:45,096 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,46781,1689422741293 (carryingMeta=false) jenkins-hbase4.apache.org,46781,1689422741293/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@189488ac[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-15 12:05:45,099 INFO [PEWorker-4] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,46781,1689422741293, splitWal=true, meta=false 2023-07-15 12:05:45,100 INFO [PEWorker-4] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,46781,1689422741293 had 1 regions 2023-07-15 12:05:45,102 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,46781,1689422741293, splitWal=true, meta=false, isMeta: false 2023-07-15 12:05:45,104 DEBUG [PEWorker-4] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293-splitting 2023-07-15 12:05:45,106 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,46781,1689422741293 WAL count=1, meta=false 2023-07-15 12:05:45,109 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660}] 2023-07-15 12:05:45,117 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:45,119 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660, worker=jenkins-hbase4.apache.org,37725,1689422741247}] 2023-07-15 12:05:45,283 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=37725] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-15 12:05:45,305 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293-splitting/jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660, size=1.1 K (1148bytes) 2023-07-15 12:05:45,306 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293-splitting/jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660 2023-07-15 12:05:45,310 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293-splitting/jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660 after 4ms 2023-07-15 12:05:45,316 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293-splitting/jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660 took 12ms 2023-07-15 12:05:45,330 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for e58b75ac0a2024c3a9f998ec952af1bd: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-15 12:05:45,332 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293-splitting/jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660 so closing down 2023-07-15 12:05:45,332 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 12:05:45,339 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-15 12:05:45,341 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660.temp 2023-07-15 12:05:45,343 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 12:05:45,358 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-15 12:05:45,360 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660.temp to hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/recovered.edits/0000000000000000013 2023-07-15 12:05:45,360 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 43 ms; skipped=1; WAL=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293-splitting/jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660, size=1.1 K, length=1148, corrupted=false, cancelled=false 2023-07-15 12:05:45,360 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293-splitting/jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660, journal: Splitting hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293-splitting/jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660, size=1.1 K (1148bytes) at 1689422745305Finishing writing output for hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293-splitting/jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660 so closing down at 1689422745332 (+27 ms)Creating recovered edits writer path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660.temp at 1689422745341 (+9 ms)3 split writer threads finished at 1689422745343 (+2 ms)Closed recovered edits writer path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689422745358 (+15 ms)Rename recovered edits hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660.temp to hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/recovered.edits/0000000000000000013 at 1689422745360 (+2 ms)Processed 11 edits across 1 Regions in 43 ms; skipped=1; WAL=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293-splitting/jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660, size=1.1 K, length=1148, corrupted=false, cancelled=false at 1689422745360 2023-07-15 12:05:45,361 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293-splitting/jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660 2023-07-15 12:05:45,363 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-15 12:05:45,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45429] master.HMaster(3641): Remote procedure done, pid=14 2023-07-15 12:05:45,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45429] wal.WALSplitUtil(141): Moved hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293-splitting/jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660 to hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/oldWALs 2023-07-15 12:05:45,372 DEBUG [hconnection-0x31a3e955-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:46781 this server is in the failed servers list 2023-07-15 12:05:45,377 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-15 12:05:45,378 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660, worker=jenkins-hbase4.apache.org,37725,1689422741247 in 255 msec 2023-07-15 12:05:45,379 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:45,383 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-15 12:05:45,383 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C46781%2C1689422741293.1689422742660, worker=jenkins-hbase4.apache.org,37725,1689422741247 in 272 msec 2023-07-15 12:05:45,385 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293-splitting dir is empty, no logs to split. 2023-07-15 12:05:45,385 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,46781,1689422741293 WAL count=0, meta=false 2023-07-15 12:05:45,385 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,46781,1689422741293 WAL splitting is done? wals=0, meta=false 2023-07-15 12:05:45,388 INFO [PEWorker-4] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,46781,1689422741293 failed, ignore...File hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,46781,1689422741293-splitting does not exist. 2023-07-15 12:05:45,390 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=e58b75ac0a2024c3a9f998ec952af1bd, ASSIGN}] 2023-07-15 12:05:45,391 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=e58b75ac0a2024c3a9f998ec952af1bd, ASSIGN 2023-07-15 12:05:45,392 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=e58b75ac0a2024c3a9f998ec952af1bd, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 12:05:45,500 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 12:05:45,543 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=e58b75ac0a2024c3a9f998ec952af1bd, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:45,543 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689422745543"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689422745543"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689422745543"}]},"ts":"1689422745543"} 2023-07-15 12:05:45,546 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure e58b75ac0a2024c3a9f998ec952af1bd, server=jenkins-hbase4.apache.org,37725,1689422741247}] 2023-07-15 12:05:45,704 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:45,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e58b75ac0a2024c3a9f998ec952af1bd, NAME => 't,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.', STARTKEY => '', ENDKEY => ''} 2023-07-15 12:05:45,705 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:45,705 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 12:05:45,705 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:45,705 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:45,707 INFO [StoreOpener-e58b75ac0a2024c3a9f998ec952af1bd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:45,708 DEBUG [StoreOpener-e58b75ac0a2024c3a9f998ec952af1bd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/f 2023-07-15 12:05:45,708 DEBUG [StoreOpener-e58b75ac0a2024c3a9f998ec952af1bd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/f 2023-07-15 12:05:45,709 INFO [StoreOpener-e58b75ac0a2024c3a9f998ec952af1bd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e58b75ac0a2024c3a9f998ec952af1bd columnFamilyName f 2023-07-15 12:05:45,709 INFO [StoreOpener-e58b75ac0a2024c3a9f998ec952af1bd-1] regionserver.HStore(310): Store=e58b75ac0a2024c3a9f998ec952af1bd/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 12:05:45,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:45,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:45,715 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/recovered.edits/0000000000000000013 2023-07-15 12:05:45,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/recovered.edits/0000000000000000013 2023-07-15 12:05:45,723 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e58b75ac0a2024c3a9f998ec952af1bd 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-15 12:05:45,776 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/.tmp/f/5d1238e7a24747edafe6240da7204f26 2023-07-15 12:05:45,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/.tmp/f/5d1238e7a24747edafe6240da7204f26 as hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/f/5d1238e7a24747edafe6240da7204f26 2023-07-15 12:05:45,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/f/5d1238e7a24747edafe6240da7204f26, entries=10, sequenceid=13, filesize=4.8 K 2023-07-15 12:05:45,829 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for e58b75ac0a2024c3a9f998ec952af1bd in 106ms, sequenceid=13, compaction requested=false; wal=null 2023-07-15 12:05:45,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-15 12:05:45,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/recovered.edits/0000000000000000013 2023-07-15 12:05:45,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:05:45,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-15 12:05:45,843 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e58b75ac0a2024c3a9f998ec952af1bd; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62950900, jitterRate=-0.061958491802215576}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 12:05:45,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e58b75ac0a2024c3a9f998ec952af1bd: 2023-07-15 12:05:45,844 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd., pid=16, masterSystemTime=1689422745699 2023-07-15 12:05:45,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:45,847 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:45,848 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=e58b75ac0a2024c3a9f998ec952af1bd, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:45,848 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689422745848"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689422745848"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689422745848"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689422745848"}]},"ts":"1689422745848"} 2023-07-15 12:05:45,853 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-15 12:05:45,853 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure e58b75ac0a2024c3a9f998ec952af1bd, server=jenkins-hbase4.apache.org,37725,1689422741247 in 304 msec 2023-07-15 12:05:45,856 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-15 12:05:45,856 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,46781,1689422741293 after splitting done 2023-07-15 12:05:45,856 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=e58b75ac0a2024c3a9f998ec952af1bd, ASSIGN in 463 msec 2023-07-15 12:05:45,856 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase4.apache.org,46781,1689422741293 from processing; numProcessing=0 2023-07-15 12:05:45,858 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,46781,1689422741293, splitWal=true, meta=false in 767 msec 2023-07-15 12:05:45,897 DEBUG [Listener at localhost/36687] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e8cbdf7 to 127.0.0.1:58992 2023-07-15 12:05:45,898 DEBUG [Listener at localhost/36687] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 12:05:45,898 INFO [Listener at localhost/36687] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-15 12:05:45,911 INFO [Listener at localhost/36687] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=272 (was 295), OpenFileDescriptor=567 (was 585), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=150 (was 150), ProcessCount=170 (was 170), AvailableMemoryMB=6318 (was 6373) 2023-07-15 12:05:45,923 INFO [Listener at localhost/36687] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=272, OpenFileDescriptor=567, MaxFileDescriptor=60000, SystemLoadAverage=150, ProcessCount=170, AvailableMemoryMB=6318 2023-07-15 12:05:45,924 INFO [Listener at localhost/36687] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase4:46781 2023-07-15 12:05:45,935 INFO [Listener at localhost/36687] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 12:05:45,935 INFO [Listener at localhost/36687] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 12:05:45,935 INFO [Listener at localhost/36687] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 12:05:45,936 INFO [Listener at localhost/36687] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 12:05:45,936 INFO [Listener at localhost/36687] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 12:05:45,936 INFO [Listener at localhost/36687] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 12:05:45,936 INFO [Listener at localhost/36687] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 12:05:45,937 INFO [Listener at localhost/36687] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41097 2023-07-15 12:05:45,938 INFO [Listener at localhost/36687] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 12:05:45,938 DEBUG [Listener at localhost/36687] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 12:05:45,939 INFO [Listener at localhost/36687] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 12:05:45,940 INFO [Listener at localhost/36687] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 12:05:45,941 INFO [Listener at localhost/36687] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41097 connecting to ZooKeeper ensemble=127.0.0.1:58992 2023-07-15 12:05:45,946 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:410970x0, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 12:05:45,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:41097-0x10168dfba350008 connected 2023-07-15 12:05:45,947 DEBUG [Listener at localhost/36687] zookeeper.ZKUtil(162): regionserver:41097-0x10168dfba350008, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 12:05:45,948 DEBUG [Listener at localhost/36687] zookeeper.ZKUtil(162): regionserver:41097-0x10168dfba350008, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-15 12:05:45,949 DEBUG [Listener at localhost/36687] zookeeper.ZKUtil(164): regionserver:41097-0x10168dfba350008, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 12:05:45,950 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41097 2023-07-15 12:05:45,950 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41097 2023-07-15 12:05:45,950 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41097 2023-07-15 12:05:45,952 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41097 2023-07-15 12:05:45,952 DEBUG [Listener at localhost/36687] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41097 2023-07-15 12:05:45,954 INFO [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(951): ClusterId : 594d7891-16a6-4fa8-a646-69890365aae2 2023-07-15 12:05:45,955 DEBUG [RS:2;jenkins-hbase4:41097] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 12:05:45,957 DEBUG [RS:2;jenkins-hbase4:41097] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 12:05:45,957 DEBUG [RS:2;jenkins-hbase4:41097] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 12:05:45,959 DEBUG [RS:2;jenkins-hbase4:41097] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 12:05:45,960 DEBUG [RS:2;jenkins-hbase4:41097] zookeeper.ReadOnlyZKClient(139): Connect 0x0361f9fe to 127.0.0.1:58992 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 12:05:45,965 DEBUG [RS:2;jenkins-hbase4:41097] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@da607e2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 12:05:45,965 DEBUG [RS:2;jenkins-hbase4:41097] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@999928e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 12:05:45,974 DEBUG [RS:2;jenkins-hbase4:41097] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:41097 2023-07-15 12:05:45,974 INFO [RS:2;jenkins-hbase4:41097] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 12:05:45,974 INFO [RS:2;jenkins-hbase4:41097] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 12:05:45,974 DEBUG [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 12:05:45,975 INFO [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,45429,1689422740097 with isa=jenkins-hbase4.apache.org/172.31.14.131:41097, startcode=1689422745935 2023-07-15 12:05:45,975 DEBUG [RS:2;jenkins-hbase4:41097] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 12:05:45,979 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58005, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 12:05:45,979 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45429] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:05:45,980 DEBUG [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c 2023-07-15 12:05:45,980 DEBUG [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:43457 2023-07-15 12:05:45,980 DEBUG [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-15 12:05:45,982 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 12:05:45,983 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 12:05:45,983 DEBUG [RS:2;jenkins-hbase4:41097] zookeeper.ZKUtil(162): regionserver:41097-0x10168dfba350008, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:05:45,983 WARN [RS:2;jenkins-hbase4:41097] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 12:05:45,983 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,41097,1689422745935] 2023-07-15 12:05:45,983 INFO [RS:2;jenkins-hbase4:41097] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 12:05:45,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:05:45,984 DEBUG [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(1948): logDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:05:45,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:45,991 DEBUG [RS:2;jenkins-hbase4:41097] zookeeper.ZKUtil(162): regionserver:41097-0x10168dfba350008, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:05:45,992 DEBUG [RS:2;jenkins-hbase4:41097] zookeeper.ZKUtil(162): regionserver:41097-0x10168dfba350008, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:05:45,993 DEBUG [RS:2;jenkins-hbase4:41097] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 12:05:45,993 INFO [RS:2;jenkins-hbase4:41097] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 12:05:45,995 INFO [RS:2;jenkins-hbase4:41097] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 12:05:45,996 INFO [RS:2;jenkins-hbase4:41097] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 12:05:45,996 INFO [RS:2;jenkins-hbase4:41097] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:45,996 INFO [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 12:05:45,998 INFO [RS:2;jenkins-hbase4:41097] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:45,998 DEBUG [RS:2;jenkins-hbase4:41097] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:45,998 DEBUG [RS:2;jenkins-hbase4:41097] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:45,998 DEBUG [RS:2;jenkins-hbase4:41097] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:45,998 DEBUG [RS:2;jenkins-hbase4:41097] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:45,999 DEBUG [RS:2;jenkins-hbase4:41097] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:45,999 DEBUG [RS:2;jenkins-hbase4:41097] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 12:05:45,999 DEBUG [RS:2;jenkins-hbase4:41097] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:45,999 DEBUG [RS:2;jenkins-hbase4:41097] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:45,999 DEBUG [RS:2;jenkins-hbase4:41097] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:45,999 DEBUG [RS:2;jenkins-hbase4:41097] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 12:05:46,000 INFO [RS:2;jenkins-hbase4:41097] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:46,000 INFO [RS:2;jenkins-hbase4:41097] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:46,000 INFO [RS:2;jenkins-hbase4:41097] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:46,014 INFO [RS:2;jenkins-hbase4:41097] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 12:05:46,014 INFO [RS:2;jenkins-hbase4:41097] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41097,1689422745935-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 12:05:46,025 INFO [RS:2;jenkins-hbase4:41097] regionserver.Replication(203): jenkins-hbase4.apache.org,41097,1689422745935 started 2023-07-15 12:05:46,025 INFO [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,41097,1689422745935, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:41097, sessionid=0x10168dfba350008 2023-07-15 12:05:46,025 INFO [Listener at localhost/36687] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:41097,5,FailOnTimeoutGroup] 2023-07-15 12:05:46,025 DEBUG [RS:2;jenkins-hbase4:41097] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 12:05:46,025 INFO [Listener at localhost/36687] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,46781,1689422741293 2023-07-15 12:05:46,026 DEBUG [RS:2;jenkins-hbase4:41097] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:05:46,026 INFO [Listener at localhost/36687] client.TestScannerTimeout(177): START ************ test3686b 2023-07-15 12:05:46,026 DEBUG [RS:2;jenkins-hbase4:41097] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41097,1689422745935' 2023-07-15 12:05:46,026 DEBUG [RS:2;jenkins-hbase4:41097] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 12:05:46,026 DEBUG [RS:2;jenkins-hbase4:41097] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 12:05:46,027 DEBUG [RS:2;jenkins-hbase4:41097] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 12:05:46,027 DEBUG [RS:2;jenkins-hbase4:41097] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 12:05:46,027 DEBUG [RS:2;jenkins-hbase4:41097] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:05:46,027 DEBUG [RS:2;jenkins-hbase4:41097] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41097,1689422745935' 2023-07-15 12:05:46,027 DEBUG [RS:2;jenkins-hbase4:41097] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 12:05:46,027 DEBUG [RS:2;jenkins-hbase4:41097] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 12:05:46,028 DEBUG [RS:2;jenkins-hbase4:41097] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 12:05:46,028 INFO [RS:2;jenkins-hbase4:41097] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 12:05:46,028 INFO [RS:2;jenkins-hbase4:41097] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 12:05:46,029 DEBUG [Listener at localhost/36687] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-15 12:05:46,029 DEBUG [Listener at localhost/36687] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:05:46,029 DEBUG [Listener at localhost/36687] zookeeper.ReadOnlyZKClient(139): Connect 0x44c30efb to 127.0.0.1:58992 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 12:05:46,034 DEBUG [Listener at localhost/36687] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37cccf27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 12:05:46,036 DEBUG [hconnection-0x1748fbbb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 12:05:46,039 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57962, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 12:05:46,130 INFO [RS:2;jenkins-hbase4:41097] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41097%2C1689422745935, suffix=, logDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,41097,1689422745935, archiveDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/oldWALs, maxLogs=32 2023-07-15 12:05:46,149 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34935,DS-f75ed93b-f0b5-4f50-8266-8678f5e04d08,DISK] 2023-07-15 12:05:46,150 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40657,DS-2f5970e4-45a6-4578-aaf0-34b1d5392eec,DISK] 2023-07-15 12:05:46,153 INFO [RS:2;jenkins-hbase4:41097] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,41097,1689422745935/jenkins-hbase4.apache.org%2C41097%2C1689422745935.1689422746132 2023-07-15 12:05:46,157 DEBUG [RS:2;jenkins-hbase4:41097] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34935,DS-f75ed93b-f0b5-4f50-8266-8678f5e04d08,DISK], DatanodeInfoWithStorage[127.0.0.1:40657,DS-2f5970e4-45a6-4578-aaf0-34b1d5392eec,DISK]] 2023-07-15 12:05:48,439 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-15 12:05:48,499 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-15 12:05:48,500 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-15 12:05:50,252 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 12:05:50,252 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-15 12:06:01,045 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease 8689278340902682643 expired clientIPAndPort=172.31.14.131:57962, userName=jenkins, regionInfo=t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:06:03,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37725] regionserver.RSRpcServices(3094): Client tried to access missing scanner 8689278340902682643 2023-07-15 12:06:03,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37725] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:57962 deadline: 1689422823046, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '8689278340902682643'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-15 12:06:03,048 DEBUG [Listener at localhost/36687] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '8689278340902682643'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '8689278340902682643'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '8689278340902682643'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-15 12:06:03,052 DEBUG [Listener at localhost/36687] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44c30efb to 127.0.0.1:58992 2023-07-15 12:06:03,052 DEBUG [Listener at localhost/36687] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 12:06:03,052 INFO [Listener at localhost/36687] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-15 12:06:03,067 INFO [Listener at localhost/36687] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=295 (was 272) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/cluster_dc5c5ec5-5294-def2-fc28-e675bb2dd0fd/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41097 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS:2;jenkins-hbase4:41097-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-177007751-172.31.14.131-1689422737126:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58992@0x0361f9fe sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/998995059.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41097 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=41097 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-798996683_17 at /127.0.0.1:34500 [Receiving block BP-177007751-172.31.14.131-1689422737126:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=41097 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: jenkins-hbase4:41097Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c-prefix:jenkins-hbase4.apache.org,41097,1689422745935 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-177007751-172.31.14.131-1689422737126:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58992@0x44c30efb-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58992@0x44c30efb-SendThread(127.0.0.1:58992) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58992@0x0361f9fe-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/cluster_dc5c5ec5-5294-def2-fc28-e675bb2dd0fd/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=41097 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=41097 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58992@0x44c30efb java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/998995059.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:41097 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58992@0x0361f9fe-SendThread(127.0.0.1:58992) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-798996683_17 at /127.0.0.1:58330 [Receiving block BP-177007751-172.31.14.131-1689422737126:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:43457 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41097 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=41097 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=41097 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=41097 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) - Thread LEAK? -, OpenFileDescriptor=571 (was 567) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=117 (was 150), ProcessCount=170 (was 170), AvailableMemoryMB=7175 (was 6318) - AvailableMemoryMB LEAK? - 2023-07-15 12:06:03,080 INFO [Listener at localhost/36687] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=295, OpenFileDescriptor=573, MaxFileDescriptor=60000, SystemLoadAverage=117, ProcessCount=170, AvailableMemoryMB=7172 2023-07-15 12:06:03,081 INFO [Listener at localhost/36687] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase4:46781 2023-07-15 12:06:03,081 INFO [Listener at localhost/36687] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,46781,1689422741293 2023-07-15 12:06:03,081 INFO [Listener at localhost/36687] client.TestScannerTimeout(104): START************ test2772 2023-07-15 12:06:03,085 DEBUG [Listener at localhost/36687] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-15 12:06:03,085 DEBUG [Listener at localhost/36687] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:06:03,085 DEBUG [Listener at localhost/36687] zookeeper.ReadOnlyZKClient(139): Connect 0x003d8b32 to 127.0.0.1:58992 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 12:06:03,092 DEBUG [Listener at localhost/36687] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d6e31d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 12:06:03,092 ERROR [Listener at localhost/36687] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,37725,1689422741247: die! ***** 2023-07-15 12:06:03,092 ERROR [Listener at localhost/36687] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-15 12:06:03,092 DEBUG [Listener at localhost/36687] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-15 12:06:03,093 DEBUG [Listener at localhost/36687] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-15 12:06:03,099 DEBUG [Listener at localhost/36687] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-15 12:06:03,101 DEBUG [Listener at localhost/36687] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-15 12:06:03,120 INFO [Listener at localhost/36687] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 983040000, "init": 513802240, "max": 2051014656, "used": 255869296 }, "NonHeapMemoryUsage": { "committed": 122314752, "init": 2555904, "max": -1, "used": 119872160 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 3, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 11957, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 61, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 61, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 61, "TotalCallTime_min": 1, "TotalCallTime_max": 1, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 61, "ResponseSize_min": 196, "ResponseSize_max": 678, "ResponseSize_mean": 437, "ResponseSize_25th_percentile": 196, "ResponseSize_median": 196, "ResponseSize_75th_percentile": 675, "ResponseSize_90th_percentile": 677, "ResponseSize_95th_percentile": 677, "ResponseSize_98th_percentile": 677, "ResponseSize_99th_percentile": 677, "ResponseSize_99.9th_percentile": 677, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 4, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 61, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16662 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:58992", "tag.serverName": "jenkins-hbase4.apache.org,37725,1689422741247", "tag.clusterId": "594d7891-16a6-4fa8-a646-69890365aae2", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 1, "maxStoreFileCount": 1, "memStoreSize": 3306, "storeFileSize": 4926, "maxStoreFileAge": 16687, "minStoreFileAge": 16687, "avgStoreFileAge": 16687, "numReferenceFiles": 0, "regionServerStartTime": 1689422741247, "averageRegionSize": 2744, "storeFileIndexSize": 320, "staticIndexSize": 31, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 546, "localBytesRead": 546, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803016, "blockCacheCount": 1, "blockCacheDataBlockCount": 1, "blockCacheSize": 602872, "blockCacheCountHitPercent": 66.66666666666666, "blockCacheExpressHitPercent": 66.66666666666666, "l1CacheSize": 602872, "l1CacheFreeSize": 819803016, "l1CacheCount": 1, "l1CacheEvictionCount": 0, "l1CacheHitCount": 2, "l1CacheMissCount": 1, "l1CacheHitRatio": 0.6666666666666666, "l1CacheMissRatio": 0.3333333333333333, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 1640, "ByteBuffAllocatorPoolAllocationBytes": 1863680, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 46, "totalRowActionRequestCount": 57, "readRequestCount": 43, "filteredReadRequestCount": 0, "writeRequestCount": 14, "rpcGetRequestCount": 10, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 22, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 14, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 2, "blockCacheHitCountPrimary": 2, "blockCacheHitCachingCount": 2, "blockCacheMissCount": 1, "blockCacheMissCountPrimary": 1, "blockCacheMissCachingCount": 1, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 1, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 2, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 10, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 270, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 21, "ScanTime_min": 0, "ScanTime_max": 1, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 14, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 1, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 10, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 4926, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 1, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 21, "ScanSize_min": 155, "ScanSize_max": 653, "ScanSize_mean": 404, "ScanSize_25th_percentile": 156, "ScanSize_median": 157, "ScanSize_75th_percentile": 652, "ScanSize_90th_percentile": 652, "ScanSize_95th_percentile": 652, "ScanSize_98th_percentile": 652, "ScanSize_99th_percentile": 652, "ScanSize_99.9th_percentile": 652, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 270, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 1, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.616790412040194, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 24, "ServerWriteQueryPerSecond_mean_rate": 1.163017895530569, "ServerWriteQueryPerSecond_1min_rate": 3.738243758742744, "ServerWriteQueryPerSecond_5min_rate": 4.565901237603426, "ServerWriteQueryPerSecond_15min_rate": 4.720662978343764 } ] } 2023-07-15 12:06:03,121 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45429] master.MasterRpcServices(612): jenkins-hbase4.apache.org,37725,1689422741247 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,37725,1689422741247: die! ***** 2023-07-15 12:06:03,122 INFO [Listener at localhost/36687] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,37725,1689422741247' ***** 2023-07-15 12:06:03,123 INFO [Listener at localhost/36687] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-15 12:06:03,124 INFO [RS:0;jenkins-hbase4:37725] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 12:06:03,124 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 12:06:03,124 INFO [RS:0;jenkins-hbase4:37725] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-15 12:06:03,124 INFO [RS:0;jenkins-hbase4:37725] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-15 12:06:03,125 INFO [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(3305): Received CLOSE for b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:06:03,126 INFO [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(3305): Received CLOSE for e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:06:03,127 INFO [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:06:03,127 DEBUG [RS:0;jenkins-hbase4:37725] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18ddedc4 to 127.0.0.1:58992 2023-07-15 12:06:03,127 DEBUG [RS:0;jenkins-hbase4:37725] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 12:06:03,127 INFO [RS:0;jenkins-hbase4:37725] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-15 12:06:03,127 INFO [RS:0;jenkins-hbase4:37725] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-15 12:06:03,127 INFO [RS:0;jenkins-hbase4:37725] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-15 12:06:03,127 INFO [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-15 12:06:03,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b84ff22b66b7d5e955328627d26038b4, disabling compactions & flushes 2023-07-15 12:06:03,128 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:06:03,128 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:06:03,128 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. after waiting 0 ms 2023-07-15 12:06:03,128 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:06:03,128 INFO [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-15 12:06:03,128 DEBUG [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(1478): Online Regions={b84ff22b66b7d5e955328627d26038b4=hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4., e58b75ac0a2024c3a9f998ec952af1bd=t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd., 1588230740=hbase:meta,,1.1588230740} 2023-07-15 12:06:03,129 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 12:06:03,129 DEBUG [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(1504): Waiting on 1588230740, b84ff22b66b7d5e955328627d26038b4, e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:06:03,129 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 12:06:03,129 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:06:03,129 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 12:06:03,131 DEBUG [hconnection-0x1927b04e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 12:06:03,131 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:06:03,131 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 12:06:03,131 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b84ff22b66b7d5e955328627d26038b4: 2023-07-15 12:06:03,131 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 12:06:03,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:06:03,133 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e58b75ac0a2024c3a9f998ec952af1bd, disabling compactions & flushes 2023-07-15 12:06:03,133 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:06:03,133 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:06:03,133 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. after waiting 0 ms 2023-07-15 12:06:03,133 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:06:03,135 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 3228 in region hbase:meta,,1.1588230740 2023-07-15 12:06:03,135 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 12:06:03,137 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 12:06:03,137 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 12:06:03,138 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-15 12:06:03,138 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32856, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 12:06:03,139 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=37725] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,37725,1689422741247 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 12:06:03,139 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:06:03,139 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=37725] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:32856 deadline: 1689422823138, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,37725,1689422741247 aborting 2023-07-15 12:06:03,139 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e58b75ac0a2024c3a9f998ec952af1bd: 2023-07-15 12:06:03,139 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:06:03,205 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 12:06:03,243 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=37725] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,37725,1689422741247 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 12:06:03,243 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=37725] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:32856 deadline: 1689422823243, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,37725,1689422741247 aborting 2023-07-15 12:06:03,329 INFO [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,37725,1689422741247; all regions closed. 2023-07-15 12:06:03,357 DEBUG [RS:0;jenkins-hbase4:37725] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 12:06:03,358 INFO [RS:0;jenkins-hbase4:37725] regionserver.LeaseManager(133): Closed leases 2023-07-15 12:06:03,358 INFO [RS:0;jenkins-hbase4:37725] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-15 12:06:03,359 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 12:06:03,360 INFO [RS:0;jenkins-hbase4:37725] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37725 2023-07-15 12:06:03,364 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:41097-0x10168dfba350008, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:06:03,364 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 12:06:03,364 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:41097-0x10168dfba350008, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 12:06:03,364 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37725,1689422741247 2023-07-15 12:06:03,365 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 12:06:03,367 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,37725,1689422741247] 2023-07-15 12:06:03,367 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,37725,1689422741247; numProcessing=1 2023-07-15 12:06:03,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:41097-0x10168dfba350008, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:06:03,368 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,37725,1689422741247 znode expired, triggering replicatorRemoved event 2023-07-15 12:06:03,369 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,37725,1689422741247 already deleted, retry=false 2023-07-15 12:06:03,369 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,37725,1689422741247 on jenkins-hbase4.apache.org,45429,1689422740097 2023-07-15 12:06:03,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:41097-0x10168dfba350008, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:06:03,371 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=17, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,37725,1689422741247, splitWal=true, meta=true 2023-07-15 12:06:03,371 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=17 for jenkins-hbase4.apache.org,37725,1689422741247 (carryingMeta=true) jenkins-hbase4.apache.org,37725,1689422741247/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@2d06a2da[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-15 12:06:03,374 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=17, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,37725,1689422741247, splitWal=true, meta=true 2023-07-15 12:06:03,375 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,37725,1689422741247, splitWal=true, meta=true, isMeta: true 2023-07-15 12:06:03,392 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting 2023-07-15 12:06:03,395 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,37725,1689422741247 WAL count=1, meta=true 2023-07-15 12:06:03,395 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta}] 2023-07-15 12:06:03,399 DEBUG [PEWorker-2] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:06:03,400 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta, worker=jenkins-hbase4.apache.org,41097,1689422745935}] 2023-07-15 12:06:03,447 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:37725 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:37725 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 12:06:03,448 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:37725 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:37725 2023-07-15 12:06:03,468 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 12:06:03,468 INFO [RS:0;jenkins-hbase4:37725] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,37725,1689422741247; zookeeper connection closed. 2023-07-15 12:06:03,468 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:37725-0x10168dfba350001, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 12:06:03,468 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2420a3f1] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2420a3f1 2023-07-15 12:06:03,553 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:06:03,553 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 12:06:03,555 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55176, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 12:06:03,555 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=41097] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=19 2023-07-15 12:06:03,574 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta, size=4.0 K (4123bytes) 2023-07-15 12:06:03,574 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta 2023-07-15 12:06:03,575 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta after 1ms 2023-07-15 12:06:03,580 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta took 6ms 2023-07-15 12:06:03,585 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-15 12:06:03,587 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta so closing down 2023-07-15 12:06:03,587 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 12:06:03,591 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta.temp 2023-07-15 12:06:03,592 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 12:06:03,611 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) 2023-07-15 12:06:03,614 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta.temp to hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-15 12:06:03,614 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 13 edits across 1 Regions in 34 ms; skipped=1; WAL=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta, size=4.0 K, length=4123, corrupted=false, cancelled=false 2023-07-15 12:06:03,614 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta, journal: Splitting hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta, size=4.0 K (4123bytes) at 1689422763574Finishing writing output for hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta so closing down at 1689422763587 (+13 ms)Creating recovered edits writer path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta.temp at 1689422763591 (+4 ms)3 split writer threads finished at 1689422763592 (+1 ms)Closed recovered edits writer path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) at 1689422763611 (+19 ms)Rename recovered edits hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta.temp to hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 at 1689422763614 (+3 ms)Processed 13 edits across 1 Regions in 34 ms; skipped=1; WAL=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta, size=4.0 K, length=4123, corrupted=false, cancelled=false at 1689422763614 2023-07-15 12:06:03,614 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta 2023-07-15 12:06:03,614 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=19 2023-07-15 12:06:03,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45429] master.HMaster(3641): Remote procedure done, pid=19 2023-07-15 12:06:03,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45429] wal.WALSplitUtil(141): Moved hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta to hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/oldWALs 2023-07-15 12:06:03,623 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=18 2023-07-15 12:06:03,623 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta, worker=jenkins-hbase4.apache.org,41097,1689422745935 in 220 msec 2023-07-15 12:06:03,624 DEBUG [PEWorker-5] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:06:03,627 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-15 12:06:03,627 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C37725%2C1689422741247.meta.1689422742856.meta, worker=jenkins-hbase4.apache.org,41097,1689422745935 in 229 msec 2023-07-15 12:06:03,638 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting dir is empty, no logs to split. 2023-07-15 12:06:03,638 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,37725,1689422741247 WAL count=0, meta=true 2023-07-15 12:06:03,638 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,37725,1689422741247 WAL splitting is done? wals=0, meta=true 2023-07-15 12:06:03,640 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-15 12:06:03,641 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-15 12:06:03,642 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 12:06:03,751 DEBUG [hconnection-0x1927b04e-metaLookup-shared--pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:37725 this server is in the failed servers list 2023-07-15 12:06:03,793 DEBUG [jenkins-hbase4:45429] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-15 12:06:03,793 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41097,1689422745935, state=OPENING 2023-07-15 12:06:03,795 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 12:06:03,795 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 12:06:03,795 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41097,1689422745935}] 2023-07-15 12:06:03,955 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-15 12:06:03,955 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 12:06:03,958 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41097%2C1689422745935.meta, suffix=.meta, logDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,41097,1689422745935, archiveDir=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/oldWALs, maxLogs=32 2023-07-15 12:06:03,977 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34935,DS-f75ed93b-f0b5-4f50-8266-8678f5e04d08,DISK] 2023-07-15 12:06:03,978 DEBUG [RS-EventLoopGroup-6-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40657,DS-2f5970e4-45a6-4578-aaf0-34b1d5392eec,DISK] 2023-07-15 12:06:03,981 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,41097,1689422745935/jenkins-hbase4.apache.org%2C41097%2C1689422745935.meta.1689422763960.meta 2023-07-15 12:06:03,981 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34935,DS-f75ed93b-f0b5-4f50-8266-8678f5e04d08,DISK], DatanodeInfoWithStorage[127.0.0.1:40657,DS-2f5970e4-45a6-4578-aaf0-34b1d5392eec,DISK]] 2023-07-15 12:06:03,982 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-15 12:06:03,982 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 12:06:03,982 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-15 12:06:03,983 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-15 12:06:03,983 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-15 12:06:03,983 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 12:06:03,983 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-15 12:06:03,983 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-15 12:06:03,985 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 12:06:03,989 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/info 2023-07-15 12:06:03,989 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/info 2023-07-15 12:06:03,990 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 12:06:03,991 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 12:06:03,991 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 12:06:03,999 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/rep_barrier 2023-07-15 12:06:03,999 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/rep_barrier 2023-07-15 12:06:04,000 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 12:06:04,001 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 12:06:04,001 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 12:06:04,003 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/table 2023-07-15 12:06:04,003 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/table 2023-07-15 12:06:04,004 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 12:06:04,005 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 12:06:04,007 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740 2023-07-15 12:06:04,010 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740 2023-07-15 12:06:04,011 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-15 12:06:04,019 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 32, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=15, path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-15 12:06:04,020 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=3.15 KB heapSize=6.27 KB 2023-07-15 12:06:04,041 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.00 KB at sequenceid=15 (bloomFilter=false), to=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/.tmp/info/10764efd613a4b6ab693f1d5077d63c6 2023-07-15 12:06:04,066 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=15 (bloomFilter=false), to=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/.tmp/table/67dba27c6fdc40adac047b10d26dc1db 2023-07-15 12:06:04,075 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/.tmp/info/10764efd613a4b6ab693f1d5077d63c6 as hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/info/10764efd613a4b6ab693f1d5077d63c6 2023-07-15 12:06:04,083 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/info/10764efd613a4b6ab693f1d5077d63c6, entries=28, sequenceid=15, filesize=7.7 K 2023-07-15 12:06:04,085 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/.tmp/table/67dba27c6fdc40adac047b10d26dc1db as hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/table/67dba27c6fdc40adac047b10d26dc1db 2023-07-15 12:06:04,092 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/table/67dba27c6fdc40adac047b10d26dc1db, entries=4, sequenceid=15, filesize=4.7 K 2023-07-15 12:06:04,093 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.15 KB/3228, heapSize ~5.98 KB/6128, currentSize=0 B/0 for 1588230740 in 73ms, sequenceid=15, compaction requested=false; wal=null 2023-07-15 12:06:04,094 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-15 12:06:04,097 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 12:06:04,099 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 12:06:04,102 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=1 2023-07-15 12:06:04,104 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61124046, jitterRate=-0.08918073773384094}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 12:06:04,104 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 12:06:04,111 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=21, masterSystemTime=1689422763948 2023-07-15 12:06:04,114 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-15 12:06:04,114 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-15 12:06:04,115 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41097,1689422745935, state=OPEN 2023-07-15 12:06:04,116 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 12:06:04,116 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 12:06:04,119 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-15 12:06:04,119 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41097,1689422745935 in 321 msec 2023-07-15 12:06:04,122 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=17 2023-07-15 12:06:04,122 INFO [PEWorker-2] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,37725,1689422741247 had 3 regions 2023-07-15 12:06:04,122 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 479 msec 2023-07-15 12:06:04,123 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,37725,1689422741247, splitWal=true, meta=true, isMeta: false 2023-07-15 12:06:04,125 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,37725,1689422741247 WAL count=1, meta=false 2023-07-15 12:06:04,125 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660}] 2023-07-15 12:06:04,127 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:06:04,128 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660, worker=jenkins-hbase4.apache.org,41097,1689422745935}] 2023-07-15 12:06:04,254 DEBUG [hconnection-0x1927b04e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 12:06:04,258 INFO [RS-EventLoopGroup-6-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55188, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 12:06:04,266 DEBUG [hconnection-0x1927b04e-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:37725 this server is in the failed servers list 2023-07-15 12:06:04,281 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=41097] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=23 2023-07-15 12:06:04,296 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660, size=899 (899bytes) 2023-07-15 12:06:04,296 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660 2023-07-15 12:06:04,297 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660 after 1ms 2023-07-15 12:06:04,302 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660 took 6ms 2023-07-15 12:06:04,309 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for b84ff22b66b7d5e955328627d26038b4: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-15 12:06:04,312 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for e58b75ac0a2024c3a9f998ec952af1bd: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-15 12:06:04,312 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660 so closing down 2023-07-15 12:06:04,312 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 12:06:04,314 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660.temp 2023-07-15 12:06:04,315 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 12:06:04,329 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-15 12:06:04,331 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660.temp to hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/recovered.edits/0000000000000000005 2023-07-15 12:06:04,331 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 4 edits across 1 Regions in 26 ms; skipped=2; WAL=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660, size=899, length=899, corrupted=false, cancelled=false 2023-07-15 12:06:04,331 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660, journal: Splitting hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660, size=899 (899bytes) at 1689422764296Finishing writing output for hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660 so closing down at 1689422764312 (+16 ms)Creating recovered edits writer path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660.temp at 1689422764315 (+3 ms)3 split writer threads finished at 1689422764315Closed recovered edits writer path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689422764329 (+14 ms)Rename recovered edits hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660.temp to hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/recovered.edits/0000000000000000005 at 1689422764331 (+2 ms)Processed 4 edits across 1 Regions in 26 ms; skipped=2; WAL=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660, size=899, length=899, corrupted=false, cancelled=false at 1689422764331 2023-07-15 12:06:04,331 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660 2023-07-15 12:06:04,331 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=23 2023-07-15 12:06:04,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45429] master.HMaster(3641): Remote procedure done, pid=23 2023-07-15 12:06:04,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45429] wal.WALSplitUtil(141): Moved hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting/jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660 to hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/oldWALs 2023-07-15 12:06:04,340 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-15 12:06:04,340 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660, worker=jenkins-hbase4.apache.org,41097,1689422745935 in 209 msec 2023-07-15 12:06:04,342 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:06:04,345 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=17 2023-07-15 12:06:04,345 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C37725%2C1689422741247.1689422742660, worker=jenkins-hbase4.apache.org,41097,1689422745935 in 217 msec 2023-07-15 12:06:04,347 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/WALs/jenkins-hbase4.apache.org,37725,1689422741247-splitting dir is empty, no logs to split. 2023-07-15 12:06:04,347 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,37725,1689422741247 WAL count=0, meta=false 2023-07-15 12:06:04,347 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,37725,1689422741247 WAL splitting is done? wals=0, meta=false 2023-07-15 12:06:04,349 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=e58b75ac0a2024c3a9f998ec952af1bd, ASSIGN}, {pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=b84ff22b66b7d5e955328627d26038b4, ASSIGN}] 2023-07-15 12:06:04,350 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=e58b75ac0a2024c3a9f998ec952af1bd, ASSIGN 2023-07-15 12:06:04,350 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=b84ff22b66b7d5e955328627d26038b4, ASSIGN 2023-07-15 12:06:04,356 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=e58b75ac0a2024c3a9f998ec952af1bd, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 12:06:04,356 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=b84ff22b66b7d5e955328627d26038b4, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 12:06:04,356 DEBUG [jenkins-hbase4:45429] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-15 12:06:04,357 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=b84ff22b66b7d5e955328627d26038b4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:06:04,357 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689422764357"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689422764357"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689422764357"}]},"ts":"1689422764357"} 2023-07-15 12:06:04,358 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=e58b75ac0a2024c3a9f998ec952af1bd, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:06:04,358 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689422764358"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689422764358"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689422764358"}]},"ts":"1689422764358"} 2023-07-15 12:06:04,358 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:37725 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:37725 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 12:06:04,360 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:37725 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:37725 2023-07-15 12:06:04,360 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:37725 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:37725 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 12:06:04,361 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:37725 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:37725 2023-07-15 12:06:04,376 DEBUG [hconnection-0x1927b04e-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:37725 this server is in the failed servers list 2023-07-15 12:06:04,462 DEBUG [PEWorker-5] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 12:06:04,463 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55190, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 12:06:04,465 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; OpenRegionProcedure b84ff22b66b7d5e955328627d26038b4, server=jenkins-hbase4.apache.org,41097,1689422745935}] 2023-07-15 12:06:04,466 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=24, state=RUNNABLE; OpenRegionProcedure e58b75ac0a2024c3a9f998ec952af1bd, server=jenkins-hbase4.apache.org,41097,1689422745935}] 2023-07-15 12:06:04,582 DEBUG [hconnection-0x1927b04e-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:37725 this server is in the failed servers list 2023-07-15 12:06:04,624 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:06:04,624 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b84ff22b66b7d5e955328627d26038b4, NAME => 'hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4.', STARTKEY => '', ENDKEY => ''} 2023-07-15 12:06:04,625 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:06:04,625 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 12:06:04,625 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:06:04,625 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:06:04,631 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:06:04,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e58b75ac0a2024c3a9f998ec952af1bd, NAME => 't,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.', STARTKEY => '', ENDKEY => ''} 2023-07-15 12:06:04,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:06:04,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 12:06:04,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:06:04,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:06:04,633 INFO [StoreOpener-b84ff22b66b7d5e955328627d26038b4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:06:04,634 DEBUG [StoreOpener-b84ff22b66b7d5e955328627d26038b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/info 2023-07-15 12:06:04,635 DEBUG [StoreOpener-b84ff22b66b7d5e955328627d26038b4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/info 2023-07-15 12:06:04,635 INFO [StoreOpener-b84ff22b66b7d5e955328627d26038b4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b84ff22b66b7d5e955328627d26038b4 columnFamilyName info 2023-07-15 12:06:04,636 INFO [StoreOpener-e58b75ac0a2024c3a9f998ec952af1bd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:06:04,636 INFO [StoreOpener-b84ff22b66b7d5e955328627d26038b4-1] regionserver.HStore(310): Store=b84ff22b66b7d5e955328627d26038b4/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 12:06:04,638 DEBUG [StoreOpener-e58b75ac0a2024c3a9f998ec952af1bd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/f 2023-07-15 12:06:04,639 DEBUG [StoreOpener-e58b75ac0a2024c3a9f998ec952af1bd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/f 2023-07-15 12:06:04,639 INFO [StoreOpener-e58b75ac0a2024c3a9f998ec952af1bd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e58b75ac0a2024c3a9f998ec952af1bd columnFamilyName f 2023-07-15 12:06:04,648 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:06:04,652 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:06:04,653 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/recovered.edits/0000000000000000005 2023-07-15 12:06:04,666 DEBUG [StoreOpener-e58b75ac0a2024c3a9f998ec952af1bd-1] regionserver.HStore(539): loaded hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/f/5d1238e7a24747edafe6240da7204f26 2023-07-15 12:06:04,666 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/recovered.edits/0000000000000000005 2023-07-15 12:06:04,667 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing b84ff22b66b7d5e955328627d26038b4 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-15 12:06:04,668 INFO [StoreOpener-e58b75ac0a2024c3a9f998ec952af1bd-1] regionserver.HStore(310): Store=e58b75ac0a2024c3a9f998ec952af1bd/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 12:06:04,669 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:06:04,671 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:06:04,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:06:04,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-15 12:06:04,688 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e58b75ac0a2024c3a9f998ec952af1bd; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64676913, jitterRate=-0.036238893866539}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 12:06:04,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e58b75ac0a2024c3a9f998ec952af1bd: 2023-07-15 12:06:04,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd., pid=27, masterSystemTime=1689422764618 2023-07-15 12:06:04,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:06:04,693 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:06:04,694 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=e58b75ac0a2024c3a9f998ec952af1bd, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:06:04,694 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689422764694"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689422764694"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689422764694"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689422764694"}]},"ts":"1689422764694"} 2023-07-15 12:06:04,701 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=24 2023-07-15 12:06:04,701 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=24, state=SUCCESS; OpenRegionProcedure e58b75ac0a2024c3a9f998ec952af1bd, server=jenkins-hbase4.apache.org,41097,1689422745935 in 231 msec 2023-07-15 12:06:04,703 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=t, region=e58b75ac0a2024c3a9f998ec952af1bd, ASSIGN in 352 msec 2023-07-15 12:06:04,713 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/.tmp/info/c0c392bc761048aab71cfcb3ddb829e3 2023-07-15 12:06:04,723 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/.tmp/info/c0c392bc761048aab71cfcb3ddb829e3 as hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/info/c0c392bc761048aab71cfcb3ddb829e3 2023-07-15 12:06:04,731 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/info/c0c392bc761048aab71cfcb3ddb829e3, entries=2, sequenceid=5, filesize=4.8 K 2023-07-15 12:06:04,731 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for b84ff22b66b7d5e955328627d26038b4 in 64ms, sequenceid=5, compaction requested=false; wal=null 2023-07-15 12:06:04,733 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/recovered.edits/0000000000000000005 2023-07-15 12:06:04,737 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:06:04,739 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-15 12:06:04,741 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b84ff22b66b7d5e955328627d26038b4; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61160899, jitterRate=-0.08863158524036407}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 12:06:04,741 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b84ff22b66b7d5e955328627d26038b4: 2023-07-15 12:06:04,748 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4., pid=26, masterSystemTime=1689422764618 2023-07-15 12:06:04,751 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:06:04,751 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:06:04,752 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=b84ff22b66b7d5e955328627d26038b4, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:06:04,752 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689422764752"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689422764752"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689422764752"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689422764752"}]},"ts":"1689422764752"} 2023-07-15 12:06:04,762 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-07-15 12:06:04,762 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; OpenRegionProcedure b84ff22b66b7d5e955328627d26038b4, server=jenkins-hbase4.apache.org,41097,1689422745935 in 294 msec 2023-07-15 12:06:04,764 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=17 2023-07-15 12:06:04,765 INFO [PEWorker-3] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,37725,1689422741247 after splitting done 2023-07-15 12:06:04,765 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=b84ff22b66b7d5e955328627d26038b4, ASSIGN in 413 msec 2023-07-15 12:06:04,765 DEBUG [PEWorker-3] master.DeadServer(114): Removed jenkins-hbase4.apache.org,37725,1689422741247 from processing; numProcessing=0 2023-07-15 12:06:04,768 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=17, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,37725,1689422741247, splitWal=true, meta=true in 1.3960 sec 2023-07-15 12:06:04,898 DEBUG [Listener at localhost/36687] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x003d8b32 to 127.0.0.1:58992 2023-07-15 12:06:04,898 DEBUG [Listener at localhost/36687] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 12:06:04,899 INFO [Listener at localhost/36687] client.TestScannerTimeout(124): END ************ test2772 2023-07-15 12:06:04,913 INFO [Listener at localhost/36687] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=278 (was 295), OpenFileDescriptor=585 (was 573) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=115 (was 117), ProcessCount=170 (was 170), AvailableMemoryMB=6943 (was 7172) 2023-07-15 12:06:04,913 INFO [Listener at localhost/36687] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-15 12:06:04,913 INFO [Listener at localhost/36687] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-15 12:06:04,913 DEBUG [Listener at localhost/36687] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4789b34a to 127.0.0.1:58992 2023-07-15 12:06:04,914 DEBUG [Listener at localhost/36687] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 12:06:04,914 DEBUG [Listener at localhost/36687] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-15 12:06:04,914 DEBUG [Listener at localhost/36687] util.JVMClusterUtil(257): Found active master hash=205168811, stopped=false 2023-07-15 12:06:04,914 INFO [Listener at localhost/36687] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,45429,1689422740097 2023-07-15 12:06:04,922 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-15 12:06:04,922 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:41097-0x10168dfba350008, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-15 12:06:04,922 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 12:06:04,922 INFO [Listener at localhost/36687] procedure2.ProcedureExecutor(629): Stopping 2023-07-15 12:06:04,922 DEBUG [Listener at localhost/36687] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bee9c84 to 127.0.0.1:58992 2023-07-15 12:06:04,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 12:06:04,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41097-0x10168dfba350008, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 12:06:04,923 DEBUG [Listener at localhost/36687] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 12:06:04,923 INFO [Listener at localhost/36687] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,41097,1689422745935' ***** 2023-07-15 12:06:04,923 INFO [Listener at localhost/36687] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-15 12:06:04,923 INFO [RS:2;jenkins-hbase4:41097] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 12:06:04,923 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 12:06:04,923 INFO [RS:2;jenkins-hbase4:41097] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-15 12:06:04,924 INFO [RS:2;jenkins-hbase4:41097] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-15 12:06:04,924 INFO [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(3305): Received CLOSE for b84ff22b66b7d5e955328627d26038b4 2023-07-15 12:06:04,926 INFO [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(3305): Received CLOSE for e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:06:04,926 INFO [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:06:04,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b84ff22b66b7d5e955328627d26038b4, disabling compactions & flushes 2023-07-15 12:06:04,927 DEBUG [RS:2;jenkins-hbase4:41097] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0361f9fe to 127.0.0.1:58992 2023-07-15 12:06:04,927 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:06:04,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:06:04,927 DEBUG [RS:2;jenkins-hbase4:41097] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 12:06:04,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. after waiting 0 ms 2023-07-15 12:06:04,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:06:04,927 INFO [RS:2;jenkins-hbase4:41097] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-15 12:06:04,927 INFO [RS:2;jenkins-hbase4:41097] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-15 12:06:04,927 INFO [RS:2;jenkins-hbase4:41097] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-15 12:06:04,928 INFO [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-15 12:06:04,928 INFO [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-15 12:06:04,929 DEBUG [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(1478): Online Regions={b84ff22b66b7d5e955328627d26038b4=hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4., e58b75ac0a2024c3a9f998ec952af1bd=t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd., 1588230740=hbase:meta,,1.1588230740} 2023-07-15 12:06:04,929 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 12:06:04,929 DEBUG [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(1504): Waiting on 1588230740, b84ff22b66b7d5e955328627d26038b4, e58b75ac0a2024c3a9f998ec952af1bd 2023-07-15 12:06:04,929 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 12:06:04,930 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 12:06:04,930 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 12:06:04,930 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 12:06:04,930 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-15 12:06:04,934 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/namespace/b84ff22b66b7d5e955328627d26038b4/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=5 2023-07-15 12:06:04,935 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:06:04,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b84ff22b66b7d5e955328627d26038b4: 2023-07-15 12:06:04,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689422743127.b84ff22b66b7d5e955328627d26038b4. 2023-07-15 12:06:04,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e58b75ac0a2024c3a9f998ec952af1bd, disabling compactions & flushes 2023-07-15 12:06:04,936 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:06:04,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:06:04,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. after waiting 0 ms 2023-07-15 12:06:04,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:06:04,941 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/default/t/e58b75ac0a2024c3a9f998ec952af1bd/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-15 12:06:04,942 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:06:04,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e58b75ac0a2024c3a9f998ec952af1bd: 2023-07-15 12:06:04,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689422743988.e58b75ac0a2024c3a9f998ec952af1bd. 2023-07-15 12:06:04,968 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=22 (bloomFilter=false), to=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/.tmp/info/64ab0a72dbf1406cbc4e25193be9db6b 2023-07-15 12:06:04,976 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/.tmp/info/64ab0a72dbf1406cbc4e25193be9db6b as hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/info/64ab0a72dbf1406cbc4e25193be9db6b 2023-07-15 12:06:04,983 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/info/64ab0a72dbf1406cbc4e25193be9db6b, entries=16, sequenceid=22, filesize=6.4 K 2023-07-15 12:06:04,986 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 56ms, sequenceid=22, compaction requested=false 2023-07-15 12:06:04,996 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/data/hbase/meta/1588230740/recovered.edits/25.seqid, newMaxSeqId=25, maxSeqId=15 2023-07-15 12:06:04,997 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 12:06:04,997 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 12:06:04,997 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 12:06:04,997 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-15 12:06:05,009 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 12:06:05,013 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-15 12:06:05,014 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-15 12:06:05,130 INFO [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,41097,1689422745935; all regions closed. 2023-07-15 12:06:05,137 DEBUG [RS:2;jenkins-hbase4:41097] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/oldWALs 2023-07-15 12:06:05,138 INFO [RS:2;jenkins-hbase4:41097] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C41097%2C1689422745935.meta:.meta(num 1689422763960) 2023-07-15 12:06:05,145 DEBUG [RS:2;jenkins-hbase4:41097] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/oldWALs 2023-07-15 12:06:05,145 INFO [RS:2;jenkins-hbase4:41097] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C41097%2C1689422745935:(num 1689422746132) 2023-07-15 12:06:05,145 DEBUG [RS:2;jenkins-hbase4:41097] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 12:06:05,145 INFO [RS:2;jenkins-hbase4:41097] regionserver.LeaseManager(133): Closed leases 2023-07-15 12:06:05,145 INFO [RS:2;jenkins-hbase4:41097] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-15 12:06:05,146 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 12:06:05,146 INFO [RS:2;jenkins-hbase4:41097] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41097 2023-07-15 12:06:05,151 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:41097-0x10168dfba350008, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41097,1689422745935 2023-07-15 12:06:05,151 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 12:06:05,151 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:41097-0x10168dfba350008, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 12:06:05,152 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,41097,1689422745935] 2023-07-15 12:06:05,152 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,41097,1689422745935; numProcessing=1 2023-07-15 12:06:05,155 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,41097,1689422745935 already deleted, retry=false 2023-07-15 12:06:05,155 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,41097,1689422745935 expired; onlineServers=0 2023-07-15 12:06:05,155 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,45429,1689422740097' ***** 2023-07-15 12:06:05,155 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-15 12:06:05,156 DEBUG [M:0;jenkins-hbase4:45429] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d07aa90, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 12:06:05,156 INFO [M:0;jenkins-hbase4:45429] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,45429,1689422740097 2023-07-15 12:06:05,156 INFO [M:0;jenkins-hbase4:45429] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,45429,1689422740097; all regions closed. 2023-07-15 12:06:05,156 DEBUG [M:0;jenkins-hbase4:45429] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 12:06:05,156 DEBUG [M:0;jenkins-hbase4:45429] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-15 12:06:05,156 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-15 12:06:05,156 DEBUG [M:0;jenkins-hbase4:45429] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-15 12:06:05,156 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689422742336] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689422742336,5,FailOnTimeoutGroup] 2023-07-15 12:06:05,157 INFO [M:0;jenkins-hbase4:45429] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-15 12:06:05,157 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689422742336] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689422742336,5,FailOnTimeoutGroup] 2023-07-15 12:06:05,157 INFO [M:0;jenkins-hbase4:45429] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-15 12:06:05,157 INFO [M:0;jenkins-hbase4:45429] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-15 12:06:05,157 DEBUG [M:0;jenkins-hbase4:45429] master.HMaster(1512): Stopping service threads 2023-07-15 12:06:05,157 INFO [M:0;jenkins-hbase4:45429] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-15 12:06:05,158 ERROR [M:0;jenkins-hbase4:45429] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (1608878744) connection to localhost/127.0.0.1:43457 from jenkins,5,PEWorkerGroup] 2023-07-15 12:06:05,158 INFO [M:0;jenkins-hbase4:45429] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-15 12:06:05,158 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-15 12:06:05,159 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-15 12:06:05,159 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 12:06:05,159 DEBUG [M:0;jenkins-hbase4:45429] zookeeper.ZKUtil(398): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-15 12:06:05,159 WARN [M:0;jenkins-hbase4:45429] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-15 12:06:05,159 INFO [M:0;jenkins-hbase4:45429] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-15 12:06:05,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 12:06:05,160 INFO [M:0;jenkins-hbase4:45429] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-15 12:06:05,160 DEBUG [M:0;jenkins-hbase4:45429] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-15 12:06:05,160 INFO [M:0;jenkins-hbase4:45429] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 12:06:05,160 DEBUG [M:0;jenkins-hbase4:45429] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 12:06:05,160 DEBUG [M:0;jenkins-hbase4:45429] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-15 12:06:05,160 DEBUG [M:0;jenkins-hbase4:45429] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 12:06:05,161 INFO [M:0;jenkins-hbase4:45429] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=90.80 KB heapSize=109.53 KB 2023-07-15 12:06:05,185 INFO [M:0;jenkins-hbase4:45429] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=90.80 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/839716ab71bf403f8331db96534863c0 2023-07-15 12:06:05,192 DEBUG [M:0;jenkins-hbase4:45429] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/839716ab71bf403f8331db96534863c0 as hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/839716ab71bf403f8331db96534863c0 2023-07-15 12:06:05,200 INFO [M:0;jenkins-hbase4:45429] regionserver.HStore(1080): Added hdfs://localhost:43457/user/jenkins/test-data/63db3e81-19ac-f3a6-c207-b0105506145c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/839716ab71bf403f8331db96534863c0, entries=27, sequenceid=212, filesize=8.2 K 2023-07-15 12:06:05,201 INFO [M:0;jenkins-hbase4:45429] regionserver.HRegion(2948): Finished flush of dataSize ~90.80 KB/92977, heapSize ~109.52 KB/112144, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 41ms, sequenceid=212, compaction requested=false 2023-07-15 12:06:05,203 INFO [M:0;jenkins-hbase4:45429] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 12:06:05,203 DEBUG [M:0;jenkins-hbase4:45429] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-15 12:06:05,207 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 12:06:05,207 INFO [M:0;jenkins-hbase4:45429] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-15 12:06:05,207 INFO [M:0;jenkins-hbase4:45429] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45429 2023-07-15 12:06:05,209 DEBUG [M:0;jenkins-hbase4:45429] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,45429,1689422740097 already deleted, retry=false 2023-07-15 12:06:05,402 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 12:06:05,402 INFO [M:0;jenkins-hbase4:45429] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,45429,1689422740097; zookeeper connection closed. 2023-07-15 12:06:05,402 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): master:45429-0x10168dfba350000, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 12:06:05,502 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:41097-0x10168dfba350008, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 12:06:05,502 INFO [RS:2;jenkins-hbase4:41097] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,41097,1689422745935; zookeeper connection closed. 2023-07-15 12:06:05,502 DEBUG [Listener at localhost/36687-EventThread] zookeeper.ZKWatcher(600): regionserver:41097-0x10168dfba350008, quorum=127.0.0.1:58992, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 12:06:05,503 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1f02805a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1f02805a 2023-07-15 12:06:05,503 INFO [Listener at localhost/36687] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-15 12:06:05,503 WARN [Listener at localhost/36687] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-15 12:06:05,508 INFO [Listener at localhost/36687] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-15 12:06:05,614 WARN [BP-177007751-172.31.14.131-1689422737126 heartbeating to localhost/127.0.0.1:43457] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-15 12:06:05,614 WARN [BP-177007751-172.31.14.131-1689422737126 heartbeating to localhost/127.0.0.1:43457] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-177007751-172.31.14.131-1689422737126 (Datanode Uuid 969e7c5c-c43d-4e03-bfdc-9128ceac1708) service to localhost/127.0.0.1:43457 2023-07-15 12:06:05,616 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/cluster_dc5c5ec5-5294-def2-fc28-e675bb2dd0fd/dfs/data/data3/current/BP-177007751-172.31.14.131-1689422737126] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 12:06:05,616 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/cluster_dc5c5ec5-5294-def2-fc28-e675bb2dd0fd/dfs/data/data4/current/BP-177007751-172.31.14.131-1689422737126] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 12:06:05,617 WARN [Listener at localhost/36687] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-15 12:06:05,624 INFO [Listener at localhost/36687] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-15 12:06:05,728 WARN [BP-177007751-172.31.14.131-1689422737126 heartbeating to localhost/127.0.0.1:43457] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-15 12:06:05,729 WARN [BP-177007751-172.31.14.131-1689422737126 heartbeating to localhost/127.0.0.1:43457] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-177007751-172.31.14.131-1689422737126 (Datanode Uuid 2482fd12-957d-4a8c-bdce-f56523fe2366) service to localhost/127.0.0.1:43457 2023-07-15 12:06:05,729 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/cluster_dc5c5ec5-5294-def2-fc28-e675bb2dd0fd/dfs/data/data1/current/BP-177007751-172.31.14.131-1689422737126] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 12:06:05,730 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/364044b8-a69d-fecd-b67f-9f5bb003b8af/cluster_dc5c5ec5-5294-def2-fc28-e675bb2dd0fd/dfs/data/data2/current/BP-177007751-172.31.14.131-1689422737126] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 12:06:05,770 INFO [Listener at localhost/36687] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-15 12:06:05,888 INFO [Listener at localhost/36687] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-15 12:06:05,934 INFO [Listener at localhost/36687] hbase.HBaseTestingUtility(1293): Minicluster is down