2023-07-11 21:44:00,295 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-11 21:44:00,580 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a 2023-07-11 21:44:00,594 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-11 21:44:00,595 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/cluster_fb640830-b4cd-c2e5-ade2-4bb61d87c207, deleteOnExit=true 2023-07-11 21:44:00,595 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-11 21:44:00,596 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/test.cache.data in system properties and HBase conf 2023-07-11 21:44:00,596 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/hadoop.tmp.dir in system properties and HBase conf 2023-07-11 21:44:00,597 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/hadoop.log.dir in system properties and HBase conf 2023-07-11 21:44:00,597 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-11 21:44:00,598 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-11 21:44:00,598 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-11 21:44:00,753 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-11 21:44:01,202 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-11 21:44:01,208 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-11 21:44:01,208 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-11 21:44:01,209 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-11 21:44:01,209 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-11 21:44:01,210 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-11 21:44:01,210 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-11 21:44:01,210 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-11 21:44:01,211 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-11 21:44:01,211 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-11 21:44:01,212 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/nfs.dump.dir in system properties and HBase conf 2023-07-11 21:44:01,212 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/java.io.tmpdir in system properties and HBase conf 2023-07-11 21:44:01,212 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-11 21:44:01,212 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-11 21:44:01,213 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-11 21:44:01,853 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-11 21:44:01,857 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-11 21:44:02,247 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-11 21:44:02,448 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-11 21:44:02,469 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-11 21:44:02,521 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-11 21:44:02,555 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/java.io.tmpdir/Jetty_localhost_41451_hdfs____nqhrrx/webapp 2023-07-11 21:44:02,699 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41451 2023-07-11 21:44:02,716 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-11 21:44:02,716 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-11 21:44:03,370 WARN [Listener at localhost/40431] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-11 21:44:03,496 WARN [Listener at localhost/40431] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-11 21:44:03,531 WARN [Listener at localhost/40431] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-11 21:44:03,538 INFO [Listener at localhost/40431] log.Slf4jLog(67): jetty-6.1.26 2023-07-11 21:44:03,543 INFO [Listener at localhost/40431] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/java.io.tmpdir/Jetty_localhost_33993_datanode____szb09m/webapp 2023-07-11 21:44:03,663 INFO [Listener at localhost/40431] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33993 2023-07-11 21:44:04,057 WARN [Listener at localhost/40521] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-11 21:44:04,125 WARN [Listener at localhost/40521] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-11 21:44:04,129 WARN [Listener at localhost/40521] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-11 21:44:04,132 INFO [Listener at localhost/40521] log.Slf4jLog(67): jetty-6.1.26 2023-07-11 21:44:04,140 INFO [Listener at localhost/40521] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/java.io.tmpdir/Jetty_localhost_42323_datanode____.v9dmeb/webapp 2023-07-11 21:44:04,270 INFO [Listener at localhost/40521] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42323 2023-07-11 21:44:04,291 WARN [Listener at localhost/45849] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-11 21:44:04,703 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8536d3c009eda8e8: Processing first storage report for DS-2a3066f7-c3e4-4af9-ae4c-b59ab55b4085 from datanode 099977b4-8786-41f4-a006-ec41424f7611 2023-07-11 21:44:04,705 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8536d3c009eda8e8: from storage DS-2a3066f7-c3e4-4af9-ae4c-b59ab55b4085 node DatanodeRegistration(127.0.0.1:46851, datanodeUuid=099977b4-8786-41f4-a006-ec41424f7611, infoPort=45353, infoSecurePort=0, ipcPort=40521, storageInfo=lv=-57;cid=testClusterID;nsid=896525810;c=1689111841947), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-11 21:44:04,705 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe5307dc253dee74b: Processing first storage report for DS-f2c706a2-5200-4d57-a191-b35285403eb2 from datanode 937b8060-e4c8-4564-9ee6-99d9602b96e2 2023-07-11 21:44:04,705 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe5307dc253dee74b: from storage DS-f2c706a2-5200-4d57-a191-b35285403eb2 node DatanodeRegistration(127.0.0.1:43955, datanodeUuid=937b8060-e4c8-4564-9ee6-99d9602b96e2, infoPort=36459, infoSecurePort=0, ipcPort=45849, storageInfo=lv=-57;cid=testClusterID;nsid=896525810;c=1689111841947), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-11 21:44:04,705 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8536d3c009eda8e8: Processing first storage report for DS-a3dfe736-af60-4be1-bf96-7f9736709a9c from datanode 099977b4-8786-41f4-a006-ec41424f7611 2023-07-11 21:44:04,706 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8536d3c009eda8e8: from storage DS-a3dfe736-af60-4be1-bf96-7f9736709a9c node DatanodeRegistration(127.0.0.1:46851, datanodeUuid=099977b4-8786-41f4-a006-ec41424f7611, infoPort=45353, infoSecurePort=0, ipcPort=40521, storageInfo=lv=-57;cid=testClusterID;nsid=896525810;c=1689111841947), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-11 21:44:04,706 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe5307dc253dee74b: Processing first storage report for DS-e5fce510-5610-4c15-9286-17e58d545d5a from datanode 937b8060-e4c8-4564-9ee6-99d9602b96e2 2023-07-11 21:44:04,706 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe5307dc253dee74b: from storage DS-e5fce510-5610-4c15-9286-17e58d545d5a node DatanodeRegistration(127.0.0.1:43955, datanodeUuid=937b8060-e4c8-4564-9ee6-99d9602b96e2, infoPort=36459, infoSecurePort=0, ipcPort=45849, storageInfo=lv=-57;cid=testClusterID;nsid=896525810;c=1689111841947), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-11 21:44:04,822 DEBUG [Listener at localhost/45849] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a 2023-07-11 21:44:04,904 INFO [Listener at localhost/45849] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/cluster_fb640830-b4cd-c2e5-ade2-4bb61d87c207/zookeeper_0, clientPort=60968, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/cluster_fb640830-b4cd-c2e5-ade2-4bb61d87c207/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/cluster_fb640830-b4cd-c2e5-ade2-4bb61d87c207/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-11 21:44:04,919 INFO [Listener at localhost/45849] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=60968 2023-07-11 21:44:04,928 INFO [Listener at localhost/45849] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 21:44:04,931 INFO [Listener at localhost/45849] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 21:44:05,749 INFO [Listener at localhost/45849] util.FSUtils(471): Created version file at hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d with version=8 2023-07-11 21:44:05,750 INFO [Listener at localhost/45849] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/hbase-staging 2023-07-11 21:44:06,213 INFO [Listener at localhost/45849] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-11 21:44:06,951 INFO [Listener at localhost/45849] client.ConnectionUtils(127): master/jenkins-hbase5:0 server-side Connection retries=45 2023-07-11 21:44:07,003 INFO [Listener at localhost/45849] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 21:44:07,004 INFO [Listener at localhost/45849] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 21:44:07,004 INFO [Listener at localhost/45849] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 21:44:07,005 INFO [Listener at localhost/45849] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 21:44:07,005 INFO [Listener at localhost/45849] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 21:44:07,235 INFO [Listener at localhost/45849] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 21:44:07,352 DEBUG [Listener at localhost/45849] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-11 21:44:07,497 INFO [Listener at localhost/45849] ipc.NettyRpcServer(120): Bind to /172.31.10.131:43121 2023-07-11 21:44:07,514 INFO [Listener at localhost/45849] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 21:44:07,517 INFO [Listener at localhost/45849] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 21:44:07,546 INFO [Listener at localhost/45849] zookeeper.RecoverableZooKeeper(93): Process identifier=master:43121 connecting to ZooKeeper ensemble=127.0.0.1:60968 2023-07-11 21:44:07,619 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:431210x0, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 21:44:07,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:43121-0x101c1f422730000 connected 2023-07-11 21:44:07,691 DEBUG [Listener at localhost/45849] zookeeper.ZKUtil(164): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 21:44:07,692 DEBUG [Listener at localhost/45849] zookeeper.ZKUtil(164): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 21:44:07,696 DEBUG [Listener at localhost/45849] zookeeper.ZKUtil(164): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 21:44:07,712 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43121 2023-07-11 21:44:07,712 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43121 2023-07-11 21:44:07,727 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43121 2023-07-11 21:44:07,731 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43121 2023-07-11 21:44:07,732 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43121 2023-07-11 21:44:07,739 INFO [Listener at localhost/45849] master.HMaster(444): hbase.rootdir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d, hbase.cluster.distributed=false 2023-07-11 21:44:07,847 INFO [Listener at localhost/45849] client.ConnectionUtils(127): regionserver/jenkins-hbase5:0 server-side Connection retries=45 2023-07-11 21:44:07,847 INFO [Listener at localhost/45849] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 21:44:07,847 INFO [Listener at localhost/45849] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 21:44:07,848 INFO [Listener at localhost/45849] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 21:44:07,848 INFO [Listener at localhost/45849] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 21:44:07,848 INFO [Listener at localhost/45849] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 21:44:07,853 INFO [Listener at localhost/45849] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 21:44:07,857 INFO [Listener at localhost/45849] ipc.NettyRpcServer(120): Bind to /172.31.10.131:43079 2023-07-11 21:44:07,859 INFO [Listener at localhost/45849] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-11 21:44:07,883 DEBUG [Listener at localhost/45849] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-11 21:44:07,885 INFO [Listener at localhost/45849] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 21:44:07,888 INFO [Listener at localhost/45849] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 21:44:07,891 INFO [Listener at localhost/45849] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43079 connecting to ZooKeeper ensemble=127.0.0.1:60968 2023-07-11 21:44:07,907 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:430790x0, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 21:44:07,924 DEBUG [Listener at localhost/45849] zookeeper.ZKUtil(164): regionserver:430790x0, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 21:44:07,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:43079-0x101c1f422730001 connected 2023-07-11 21:44:07,929 DEBUG [Listener at localhost/45849] zookeeper.ZKUtil(164): regionserver:43079-0x101c1f422730001, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 21:44:07,930 DEBUG [Listener at localhost/45849] zookeeper.ZKUtil(164): regionserver:43079-0x101c1f422730001, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 21:44:07,936 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43079 2023-07-11 21:44:07,943 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43079 2023-07-11 21:44:07,944 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43079 2023-07-11 21:44:07,951 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43079 2023-07-11 21:44:07,954 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43079 2023-07-11 21:44:07,969 INFO [Listener at localhost/45849] client.ConnectionUtils(127): regionserver/jenkins-hbase5:0 server-side Connection retries=45 2023-07-11 21:44:07,970 INFO [Listener at localhost/45849] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 21:44:07,970 INFO [Listener at localhost/45849] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 21:44:07,970 INFO [Listener at localhost/45849] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 21:44:07,970 INFO [Listener at localhost/45849] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 21:44:07,972 INFO [Listener at localhost/45849] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 21:44:07,972 INFO [Listener at localhost/45849] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 21:44:07,980 INFO [Listener at localhost/45849] ipc.NettyRpcServer(120): Bind to /172.31.10.131:37187 2023-07-11 21:44:07,981 INFO [Listener at localhost/45849] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-11 21:44:07,988 DEBUG [Listener at localhost/45849] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-11 21:44:07,990 INFO [Listener at localhost/45849] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 21:44:07,992 INFO [Listener at localhost/45849] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 21:44:07,995 INFO [Listener at localhost/45849] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37187 connecting to ZooKeeper ensemble=127.0.0.1:60968 2023-07-11 21:44:08,002 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:371870x0, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 21:44:08,003 DEBUG [Listener at localhost/45849] zookeeper.ZKUtil(164): regionserver:371870x0, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 21:44:08,004 DEBUG [Listener at localhost/45849] zookeeper.ZKUtil(164): regionserver:371870x0, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 21:44:08,005 DEBUG [Listener at localhost/45849] zookeeper.ZKUtil(164): regionserver:371870x0, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 21:44:08,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:37187-0x101c1f422730002 connected 2023-07-11 21:44:08,011 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37187 2023-07-11 21:44:08,012 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37187 2023-07-11 21:44:08,015 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37187 2023-07-11 21:44:08,021 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37187 2023-07-11 21:44:08,021 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37187 2023-07-11 21:44:08,040 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase5.apache.org,43121,1689111846003 2023-07-11 21:44:08,053 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 21:44:08,054 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase5.apache.org,43121,1689111846003 2023-07-11 21:44:08,081 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:43079-0x101c1f422730001, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 21:44:08,081 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 21:44:08,081 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:37187-0x101c1f422730002, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 21:44:08,081 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 21:44:08,082 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 21:44:08,083 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase5.apache.org,43121,1689111846003 from backup master directory 2023-07-11 21:44:08,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 21:44:08,089 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase5.apache.org,43121,1689111846003 2023-07-11 21:44:08,089 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 21:44:08,091 WARN [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 21:44:08,091 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase5.apache.org,43121,1689111846003 2023-07-11 21:44:08,095 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-11 21:44:08,099 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-11 21:44:08,292 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/hbase.id with ID: 5ef11920-c85b-4d5d-931f-0e6db259ed46 2023-07-11 21:44:08,354 INFO [master/jenkins-hbase5:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 21:44:08,381 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 21:44:08,494 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2e9dcc78 to 127.0.0.1:60968 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 21:44:08,571 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a2e9143, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 21:44:08,604 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-11 21:44:08,607 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-11 21:44:08,632 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-11 21:44:08,632 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-11 21:44:08,636 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-11 21:44:08,643 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-11 21:44:08,645 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 21:44:08,684 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/MasterData/data/master/store-tmp 2023-07-11 21:44:08,734 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 21:44:08,734 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 21:44:08,734 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 21:44:08,735 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 21:44:08,735 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 21:44:08,735 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 21:44:08,735 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 21:44:08,735 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 21:44:08,737 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/MasterData/WALs/jenkins-hbase5.apache.org,43121,1689111846003 2023-07-11 21:44:08,764 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C43121%2C1689111846003, suffix=, logDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/MasterData/WALs/jenkins-hbase5.apache.org,43121,1689111846003, archiveDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/MasterData/oldWALs, maxLogs=10 2023-07-11 21:44:08,833 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43955,DS-f2c706a2-5200-4d57-a191-b35285403eb2,DISK] 2023-07-11 21:44:08,833 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46851,DS-2a3066f7-c3e4-4af9-ae4c-b59ab55b4085,DISK] 2023-07-11 21:44:08,845 DEBUG [RS-EventLoopGroup-4-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-11 21:44:08,926 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/MasterData/WALs/jenkins-hbase5.apache.org,43121,1689111846003/jenkins-hbase5.apache.org%2C43121%2C1689111846003.1689111848776 2023-07-11 21:44:08,927 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46851,DS-2a3066f7-c3e4-4af9-ae4c-b59ab55b4085,DISK], DatanodeInfoWithStorage[127.0.0.1:43955,DS-f2c706a2-5200-4d57-a191-b35285403eb2,DISK]] 2023-07-11 21:44:08,927 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-11 21:44:08,928 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 21:44:08,931 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 21:44:08,932 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 21:44:09,007 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-11 21:44:09,033 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-11 21:44:09,083 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-11 21:44:09,112 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 21:44:09,121 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-11 21:44:09,124 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-11 21:44:09,157 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 21:44:09,164 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 21:44:09,169 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63730906, jitterRate=-0.05033549666404724}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 21:44:09,169 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 21:44:09,172 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-11 21:44:09,200 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-11 21:44:09,201 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-11 21:44:09,204 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 21:44:09,207 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-11 21:44:09,259 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 51 msec 2023-07-11 21:44:09,259 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-11 21:44:09,316 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-11 21:44:09,324 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-11 21:44:09,369 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-11 21:44:09,373 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-11 21:44:09,375 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-11 21:44:09,381 INFO [master/jenkins-hbase5:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-11 21:44:09,387 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-11 21:44:09,392 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 21:44:09,394 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-11 21:44:09,395 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-11 21:44:09,409 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-11 21:44:09,416 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:37187-0x101c1f422730002, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 21:44:09,416 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 21:44:09,416 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 21:44:09,417 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:43079-0x101c1f422730001, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 21:44:09,421 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase5.apache.org,43121,1689111846003, sessionid=0x101c1f422730000, setting cluster-up flag (Was=false) 2023-07-11 21:44:09,442 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 21:44:09,452 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-11 21:44:09,455 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,43121,1689111846003 2023-07-11 21:44:09,465 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 21:44:09,475 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-11 21:44:09,478 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,43121,1689111846003 2023-07-11 21:44:09,481 WARN [master/jenkins-hbase5:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/.hbase-snapshot/.tmp 2023-07-11 21:44:09,532 INFO [RS:0;jenkins-hbase5:43079] regionserver.HRegionServer(951): ClusterId : 5ef11920-c85b-4d5d-931f-0e6db259ed46 2023-07-11 21:44:09,552 DEBUG [RS:0;jenkins-hbase5:43079] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-11 21:44:09,546 INFO [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(951): ClusterId : 5ef11920-c85b-4d5d-931f-0e6db259ed46 2023-07-11 21:44:09,554 DEBUG [RS:1;jenkins-hbase5:37187] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-11 21:44:09,561 DEBUG [RS:0;jenkins-hbase5:43079] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-11 21:44:09,561 DEBUG [RS:0;jenkins-hbase5:43079] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-11 21:44:09,561 DEBUG [RS:1;jenkins-hbase5:37187] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-11 21:44:09,562 DEBUG [RS:1;jenkins-hbase5:37187] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-11 21:44:09,570 DEBUG [RS:1;jenkins-hbase5:37187] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-11 21:44:09,572 DEBUG [RS:0;jenkins-hbase5:43079] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-11 21:44:09,573 DEBUG [RS:0;jenkins-hbase5:43079] zookeeper.ReadOnlyZKClient(139): Connect 0x0c21f3da to 127.0.0.1:60968 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 21:44:09,573 DEBUG [RS:1;jenkins-hbase5:37187] zookeeper.ReadOnlyZKClient(139): Connect 0x359d0571 to 127.0.0.1:60968 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 21:44:09,624 DEBUG [RS:0;jenkins-hbase5:43079] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2e60c2e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 21:44:09,625 DEBUG [RS:0;jenkins-hbase5:43079] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6d998aa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-11 21:44:09,626 DEBUG [RS:1;jenkins-hbase5:37187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b5aedb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 21:44:09,626 DEBUG [RS:1;jenkins-hbase5:37187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f656e01, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-11 21:44:09,653 DEBUG [RS:0;jenkins-hbase5:43079] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase5:43079 2023-07-11 21:44:09,653 DEBUG [RS:1;jenkins-hbase5:37187] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase5:37187 2023-07-11 21:44:09,659 INFO [RS:0;jenkins-hbase5:43079] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-11 21:44:09,659 INFO [RS:0;jenkins-hbase5:43079] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-11 21:44:09,659 DEBUG [RS:0;jenkins-hbase5:43079] regionserver.HRegionServer(1022): About to register with Master. 2023-07-11 21:44:09,661 INFO [RS:1;jenkins-hbase5:37187] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-11 21:44:09,662 INFO [RS:1;jenkins-hbase5:37187] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-11 21:44:09,664 DEBUG [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(1022): About to register with Master. 2023-07-11 21:44:09,664 INFO [RS:0;jenkins-hbase5:43079] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase5.apache.org,43121,1689111846003 with isa=jenkins-hbase5.apache.org/172.31.10.131:43079, startcode=1689111847846 2023-07-11 21:44:09,667 INFO [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase5.apache.org,43121,1689111846003 with isa=jenkins-hbase5.apache.org/172.31.10.131:37187, startcode=1689111847969 2023-07-11 21:44:09,673 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-11 21:44:09,685 DEBUG [RS:1;jenkins-hbase5:37187] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-11 21:44:09,685 DEBUG [RS:0;jenkins-hbase5:43079] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-11 21:44:09,687 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-11 21:44:09,688 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-11 21:44:09,688 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-11 21:44:09,688 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-11 21:44:09,688 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase5:0, corePoolSize=10, maxPoolSize=10 2023-07-11 21:44:09,688 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,688 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-11 21:44:09,689 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,703 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689111879703 2023-07-11 21:44:09,706 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-11 21:44:09,709 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-11 21:44:09,709 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-11 21:44:09,719 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-11 21:44:09,720 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-11 21:44:09,728 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-11 21:44:09,731 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-11 21:44:09,732 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-11 21:44:09,732 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-11 21:44:09,736 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:09,738 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-11 21:44:09,740 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-11 21:44:09,741 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-11 21:44:09,750 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-11 21:44:09,751 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-11 21:44:09,758 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689111849755,5,FailOnTimeoutGroup] 2023-07-11 21:44:09,759 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689111849758,5,FailOnTimeoutGroup] 2023-07-11 21:44:09,759 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:09,759 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-11 21:44:09,761 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:09,761 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:09,765 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:45673, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-11 21:44:09,765 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:54755, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-11 21:44:09,785 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=43121] master.ServerManager(394): Registering regionserver=jenkins-hbase5.apache.org,43079,1689111847846 2023-07-11 21:44:09,789 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-11 21:44:09,790 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-11 21:44:09,790 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d 2023-07-11 21:44:09,801 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=43121] master.ServerManager(394): Registering regionserver=jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:09,815 DEBUG [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d 2023-07-11 21:44:09,815 DEBUG [RS:0;jenkins-hbase5:43079] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d 2023-07-11 21:44:09,816 DEBUG [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:40431 2023-07-11 21:44:09,816 DEBUG [RS:0;jenkins-hbase5:43079] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:40431 2023-07-11 21:44:09,816 DEBUG [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-11 21:44:09,816 DEBUG [RS:0;jenkins-hbase5:43079] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-11 21:44:09,838 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 21:44:09,839 DEBUG [RS:0;jenkins-hbase5:43079] zookeeper.ZKUtil(162): regionserver:43079-0x101c1f422730001, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,43079,1689111847846 2023-07-11 21:44:09,839 DEBUG [RS:1;jenkins-hbase5:37187] zookeeper.ZKUtil(162): regionserver:37187-0x101c1f422730002, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:09,839 WARN [RS:0;jenkins-hbase5:43079] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 21:44:09,839 WARN [RS:1;jenkins-hbase5:37187] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 21:44:09,843 INFO [RS:1;jenkins-hbase5:37187] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 21:44:09,843 DEBUG [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(1948): logDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:09,841 INFO [RS:0;jenkins-hbase5:43079] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 21:44:09,845 DEBUG [RS:0;jenkins-hbase5:43079] regionserver.HRegionServer(1948): logDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846 2023-07-11 21:44:09,846 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 21:44:09,845 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase5.apache.org,43079,1689111847846] 2023-07-11 21:44:09,864 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase5.apache.org,37187,1689111847969] 2023-07-11 21:44:09,890 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-11 21:44:09,894 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/info 2023-07-11 21:44:09,895 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-11 21:44:09,896 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 21:44:09,896 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-11 21:44:09,897 DEBUG [RS:0;jenkins-hbase5:43079] zookeeper.ZKUtil(162): regionserver:43079-0x101c1f422730001, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,43079,1689111847846 2023-07-11 21:44:09,898 DEBUG [RS:1;jenkins-hbase5:37187] zookeeper.ZKUtil(162): regionserver:37187-0x101c1f422730002, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,43079,1689111847846 2023-07-11 21:44:09,898 DEBUG [RS:1;jenkins-hbase5:37187] zookeeper.ZKUtil(162): regionserver:37187-0x101c1f422730002, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:09,898 DEBUG [RS:0;jenkins-hbase5:43079] zookeeper.ZKUtil(162): regionserver:43079-0x101c1f422730001, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:09,900 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/rep_barrier 2023-07-11 21:44:09,901 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-11 21:44:09,902 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 21:44:09,902 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-11 21:44:09,912 DEBUG [RS:0;jenkins-hbase5:43079] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-11 21:44:09,917 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/table 2023-07-11 21:44:09,919 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-11 21:44:09,920 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 21:44:09,912 DEBUG [RS:1;jenkins-hbase5:37187] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-11 21:44:09,942 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740 2023-07-11 21:44:09,943 INFO [RS:1;jenkins-hbase5:37187] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-11 21:44:09,943 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740 2023-07-11 21:44:09,944 INFO [RS:0;jenkins-hbase5:43079] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-11 21:44:09,948 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-11 21:44:09,951 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-11 21:44:09,960 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 21:44:09,962 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68732778, jitterRate=0.024198204278945923}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-11 21:44:09,962 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-11 21:44:09,962 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-11 21:44:09,962 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-11 21:44:09,962 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-11 21:44:09,962 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-11 21:44:09,962 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-11 21:44:09,963 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-11 21:44:09,964 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-11 21:44:09,970 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-11 21:44:09,970 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-11 21:44:09,972 INFO [RS:1;jenkins-hbase5:37187] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-11 21:44:09,972 INFO [RS:0;jenkins-hbase5:43079] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-11 21:44:09,977 INFO [RS:0;jenkins-hbase5:43079] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-11 21:44:09,977 INFO [RS:1;jenkins-hbase5:37187] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-11 21:44:09,978 INFO [RS:0;jenkins-hbase5:43079] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:09,978 INFO [RS:1;jenkins-hbase5:37187] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:09,978 INFO [RS:0;jenkins-hbase5:43079] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-11 21:44:09,979 INFO [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-11 21:44:09,981 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-11 21:44:09,988 INFO [RS:0;jenkins-hbase5:43079] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:09,988 INFO [RS:1;jenkins-hbase5:37187] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:09,988 DEBUG [RS:0;jenkins-hbase5:43079] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,988 DEBUG [RS:1;jenkins-hbase5:37187] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,989 DEBUG [RS:0;jenkins-hbase5:43079] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,989 DEBUG [RS:1;jenkins-hbase5:37187] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,989 DEBUG [RS:0;jenkins-hbase5:43079] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,989 DEBUG [RS:1;jenkins-hbase5:37187] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,989 DEBUG [RS:0;jenkins-hbase5:43079] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,989 DEBUG [RS:1;jenkins-hbase5:37187] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,990 DEBUG [RS:0;jenkins-hbase5:43079] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,990 DEBUG [RS:1;jenkins-hbase5:37187] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,990 DEBUG [RS:0;jenkins-hbase5:43079] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-11 21:44:09,990 DEBUG [RS:1;jenkins-hbase5:37187] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-11 21:44:09,990 DEBUG [RS:0;jenkins-hbase5:43079] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,990 DEBUG [RS:1;jenkins-hbase5:37187] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,990 DEBUG [RS:0;jenkins-hbase5:43079] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,990 DEBUG [RS:1;jenkins-hbase5:37187] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,990 DEBUG [RS:0;jenkins-hbase5:43079] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,990 DEBUG [RS:1;jenkins-hbase5:37187] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,991 DEBUG [RS:0;jenkins-hbase5:43079] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,991 DEBUG [RS:1;jenkins-hbase5:37187] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:09,992 INFO [RS:0;jenkins-hbase5:43079] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:09,992 INFO [RS:0;jenkins-hbase5:43079] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:09,992 INFO [RS:1;jenkins-hbase5:37187] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:09,992 INFO [RS:0;jenkins-hbase5:43079] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:09,992 INFO [RS:1;jenkins-hbase5:37187] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:09,993 INFO [RS:1;jenkins-hbase5:37187] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:09,996 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-11 21:44:10,001 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-11 21:44:10,014 INFO [RS:1;jenkins-hbase5:37187] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-11 21:44:10,014 INFO [RS:0;jenkins-hbase5:43079] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-11 21:44:10,018 INFO [RS:0;jenkins-hbase5:43079] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,43079,1689111847846-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:10,018 INFO [RS:1;jenkins-hbase5:37187] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,37187,1689111847969-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:10,043 INFO [RS:0;jenkins-hbase5:43079] regionserver.Replication(203): jenkins-hbase5.apache.org,43079,1689111847846 started 2023-07-11 21:44:10,043 INFO [RS:0;jenkins-hbase5:43079] regionserver.HRegionServer(1637): Serving as jenkins-hbase5.apache.org,43079,1689111847846, RpcServer on jenkins-hbase5.apache.org/172.31.10.131:43079, sessionid=0x101c1f422730001 2023-07-11 21:44:10,043 DEBUG [RS:0;jenkins-hbase5:43079] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-11 21:44:10,044 DEBUG [RS:0;jenkins-hbase5:43079] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase5.apache.org,43079,1689111847846 2023-07-11 21:44:10,044 DEBUG [RS:0;jenkins-hbase5:43079] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,43079,1689111847846' 2023-07-11 21:44:10,044 DEBUG [RS:0;jenkins-hbase5:43079] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-11 21:44:10,045 INFO [RS:1;jenkins-hbase5:37187] regionserver.Replication(203): jenkins-hbase5.apache.org,37187,1689111847969 started 2023-07-11 21:44:10,045 INFO [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(1637): Serving as jenkins-hbase5.apache.org,37187,1689111847969, RpcServer on jenkins-hbase5.apache.org/172.31.10.131:37187, sessionid=0x101c1f422730002 2023-07-11 21:44:10,045 DEBUG [RS:1;jenkins-hbase5:37187] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-11 21:44:10,045 DEBUG [RS:1;jenkins-hbase5:37187] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:10,046 DEBUG [RS:1;jenkins-hbase5:37187] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,37187,1689111847969' 2023-07-11 21:44:10,046 DEBUG [RS:1;jenkins-hbase5:37187] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-11 21:44:10,047 DEBUG [RS:0;jenkins-hbase5:43079] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-11 21:44:10,047 DEBUG [RS:1;jenkins-hbase5:37187] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-11 21:44:10,048 DEBUG [RS:0;jenkins-hbase5:43079] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-11 21:44:10,048 DEBUG [RS:0;jenkins-hbase5:43079] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-11 21:44:10,048 DEBUG [RS:0;jenkins-hbase5:43079] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase5.apache.org,43079,1689111847846 2023-07-11 21:44:10,048 DEBUG [RS:0;jenkins-hbase5:43079] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,43079,1689111847846' 2023-07-11 21:44:10,048 DEBUG [RS:0;jenkins-hbase5:43079] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-11 21:44:10,049 DEBUG [RS:1;jenkins-hbase5:37187] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-11 21:44:10,049 DEBUG [RS:1;jenkins-hbase5:37187] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-11 21:44:10,049 DEBUG [RS:1;jenkins-hbase5:37187] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:10,049 DEBUG [RS:1;jenkins-hbase5:37187] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,37187,1689111847969' 2023-07-11 21:44:10,049 DEBUG [RS:0;jenkins-hbase5:43079] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-11 21:44:10,049 DEBUG [RS:1;jenkins-hbase5:37187] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-11 21:44:10,050 DEBUG [RS:0;jenkins-hbase5:43079] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-11 21:44:10,050 INFO [RS:0;jenkins-hbase5:43079] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-11 21:44:10,050 DEBUG [RS:1;jenkins-hbase5:37187] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-11 21:44:10,050 INFO [RS:0;jenkins-hbase5:43079] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-11 21:44:10,051 DEBUG [RS:1;jenkins-hbase5:37187] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-11 21:44:10,051 INFO [RS:1;jenkins-hbase5:37187] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-11 21:44:10,052 INFO [RS:1;jenkins-hbase5:37187] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-11 21:44:10,154 DEBUG [jenkins-hbase5:43121] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-11 21:44:10,171 DEBUG [jenkins-hbase5:43121] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-11 21:44:10,174 INFO [RS:0;jenkins-hbase5:43079] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C43079%2C1689111847846, suffix=, logDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846, archiveDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/oldWALs, maxLogs=32 2023-07-11 21:44:10,175 INFO [RS:1;jenkins-hbase5:37187] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C37187%2C1689111847969, suffix=, logDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969, archiveDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/oldWALs, maxLogs=32 2023-07-11 21:44:10,182 DEBUG [jenkins-hbase5:43121] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-11 21:44:10,182 DEBUG [jenkins-hbase5:43121] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-11 21:44:10,182 DEBUG [jenkins-hbase5:43121] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-11 21:44:10,186 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,37187,1689111847969, state=OPENING 2023-07-11 21:44:10,194 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-11 21:44:10,196 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 21:44:10,197 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 21:44:10,201 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,37187,1689111847969}] 2023-07-11 21:44:10,217 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46851,DS-2a3066f7-c3e4-4af9-ae4c-b59ab55b4085,DISK] 2023-07-11 21:44:10,220 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43955,DS-f2c706a2-5200-4d57-a191-b35285403eb2,DISK] 2023-07-11 21:44:10,232 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43955,DS-f2c706a2-5200-4d57-a191-b35285403eb2,DISK] 2023-07-11 21:44:10,240 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46851,DS-2a3066f7-c3e4-4af9-ae4c-b59ab55b4085,DISK] 2023-07-11 21:44:10,271 INFO [RS:0;jenkins-hbase5:43079] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846/jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189 2023-07-11 21:44:10,271 INFO [RS:1;jenkins-hbase5:37187] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969/jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181 2023-07-11 21:44:10,271 DEBUG [RS:0;jenkins-hbase5:43079] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46851,DS-2a3066f7-c3e4-4af9-ae4c-b59ab55b4085,DISK], DatanodeInfoWithStorage[127.0.0.1:43955,DS-f2c706a2-5200-4d57-a191-b35285403eb2,DISK]] 2023-07-11 21:44:10,288 DEBUG [RS:1;jenkins-hbase5:37187] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43955,DS-f2c706a2-5200-4d57-a191-b35285403eb2,DISK], DatanodeInfoWithStorage[127.0.0.1:46851,DS-2a3066f7-c3e4-4af9-ae4c-b59ab55b4085,DISK]] 2023-07-11 21:44:10,423 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:10,427 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-11 21:44:10,431 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:33052, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-11 21:44:10,453 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-11 21:44:10,453 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 21:44:10,457 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta, suffix=.meta, logDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969, archiveDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/oldWALs, maxLogs=32 2023-07-11 21:44:10,480 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46851,DS-2a3066f7-c3e4-4af9-ae4c-b59ab55b4085,DISK] 2023-07-11 21:44:10,491 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43955,DS-f2c706a2-5200-4d57-a191-b35285403eb2,DISK] 2023-07-11 21:44:10,500 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969/jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta 2023-07-11 21:44:10,508 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46851,DS-2a3066f7-c3e4-4af9-ae4c-b59ab55b4085,DISK], DatanodeInfoWithStorage[127.0.0.1:43955,DS-f2c706a2-5200-4d57-a191-b35285403eb2,DISK]] 2023-07-11 21:44:10,509 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-11 21:44:10,512 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-11 21:44:10,537 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-11 21:44:10,544 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-11 21:44:10,550 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-11 21:44:10,550 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 21:44:10,550 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-11 21:44:10,550 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-11 21:44:10,555 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-11 21:44:10,558 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/info 2023-07-11 21:44:10,558 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/info 2023-07-11 21:44:10,559 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-11 21:44:10,561 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 21:44:10,561 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-11 21:44:10,563 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/rep_barrier 2023-07-11 21:44:10,563 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/rep_barrier 2023-07-11 21:44:10,564 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-11 21:44:10,567 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 21:44:10,567 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-11 21:44:10,569 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/table 2023-07-11 21:44:10,569 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/table 2023-07-11 21:44:10,569 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-11 21:44:10,570 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 21:44:10,572 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740 2023-07-11 21:44:10,577 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740 2023-07-11 21:44:10,581 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-11 21:44:10,591 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-11 21:44:10,593 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71321162, jitterRate=0.0627681314945221}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-11 21:44:10,593 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-11 21:44:10,606 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689111850413 2023-07-11 21:44:10,635 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,37187,1689111847969, state=OPEN 2023-07-11 21:44:10,638 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-11 21:44:10,639 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-11 21:44:10,641 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-11 21:44:10,641 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 21:44:10,647 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-11 21:44:10,647 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,37187,1689111847969 in 440 msec 2023-07-11 21:44:10,653 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-11 21:44:10,653 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 668 msec 2023-07-11 21:44:10,659 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.0920 sec 2023-07-11 21:44:10,660 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689111850659, completionTime=-1 2023-07-11 21:44:10,660 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-11 21:44:10,661 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 21:44:10,742 DEBUG [hconnection-0x27bbde66-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 21:44:10,745 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:33064, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 21:44:10,792 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-11 21:44:10,793 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689111910793 2023-07-11 21:44:10,793 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689111970793 2023-07-11 21:44:10,793 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 131 msec 2023-07-11 21:44:10,819 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,43121,1689111846003-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:10,819 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,43121,1689111846003-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:10,819 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,43121,1689111846003-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:10,821 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase5:43121, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:10,821 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:10,828 DEBUG [master/jenkins-hbase5:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-11 21:44:10,837 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-11 21:44:10,838 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-11 21:44:10,848 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-11 21:44:10,851 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-11 21:44:10,853 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-11 21:44:10,876 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/.tmp/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:10,879 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/.tmp/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411 empty. 2023-07-11 21:44:10,879 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/.tmp/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:10,880 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-11 21:44:10,906 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-11 21:44:10,908 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => e58a0dd97190346bf477f074c53b0411, NAME => 'hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/.tmp 2023-07-11 21:44:10,922 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 21:44:10,922 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing e58a0dd97190346bf477f074c53b0411, disabling compactions & flushes 2023-07-11 21:44:10,922 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:10,922 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:10,923 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. after waiting 0 ms 2023-07-11 21:44:10,923 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:10,923 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:10,923 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for e58a0dd97190346bf477f074c53b0411: 2023-07-11 21:44:10,928 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-11 21:44:10,943 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689111850930"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689111850930"}]},"ts":"1689111850930"} 2023-07-11 21:44:10,970 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-11 21:44:10,973 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-11 21:44:10,977 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689111850973"}]},"ts":"1689111850973"} 2023-07-11 21:44:10,983 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-11 21:44:10,987 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-11 21:44:10,989 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-11 21:44:10,989 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-11 21:44:10,989 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-11 21:44:10,992 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e58a0dd97190346bf477f074c53b0411, ASSIGN}] 2023-07-11 21:44:10,995 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e58a0dd97190346bf477f074c53b0411, ASSIGN 2023-07-11 21:44:11,000 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=e58a0dd97190346bf477f074c53b0411, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,37187,1689111847969; forceNewPlan=false, retain=false 2023-07-11 21:44:11,153 INFO [jenkins-hbase5:43121] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-11 21:44:11,154 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e58a0dd97190346bf477f074c53b0411, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:11,155 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689111851154"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689111851154"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689111851154"}]},"ts":"1689111851154"} 2023-07-11 21:44:11,165 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure e58a0dd97190346bf477f074c53b0411, server=jenkins-hbase5.apache.org,37187,1689111847969}] 2023-07-11 21:44:11,337 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:11,338 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e58a0dd97190346bf477f074c53b0411, NAME => 'hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411.', STARTKEY => '', ENDKEY => ''} 2023-07-11 21:44:11,338 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:11,338 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 21:44:11,338 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:11,338 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:11,343 INFO [StoreOpener-e58a0dd97190346bf477f074c53b0411-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:11,347 DEBUG [StoreOpener-e58a0dd97190346bf477f074c53b0411-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/info 2023-07-11 21:44:11,347 DEBUG [StoreOpener-e58a0dd97190346bf477f074c53b0411-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/info 2023-07-11 21:44:11,348 INFO [StoreOpener-e58a0dd97190346bf477f074c53b0411-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e58a0dd97190346bf477f074c53b0411 columnFamilyName info 2023-07-11 21:44:11,349 INFO [StoreOpener-e58a0dd97190346bf477f074c53b0411-1] regionserver.HStore(310): Store=e58a0dd97190346bf477f074c53b0411/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 21:44:11,354 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:11,356 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:11,362 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:11,366 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 21:44:11,367 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened e58a0dd97190346bf477f074c53b0411; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65419992, jitterRate=-0.02516615390777588}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 21:44:11,367 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for e58a0dd97190346bf477f074c53b0411: 2023-07-11 21:44:11,369 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411., pid=6, masterSystemTime=1689111851323 2023-07-11 21:44:11,376 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:11,376 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:11,380 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e58a0dd97190346bf477f074c53b0411, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:11,381 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689111851380"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689111851380"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689111851380"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689111851380"}]},"ts":"1689111851380"} 2023-07-11 21:44:11,390 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-11 21:44:11,390 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure e58a0dd97190346bf477f074c53b0411, server=jenkins-hbase5.apache.org,37187,1689111847969 in 221 msec 2023-07-11 21:44:11,395 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-11 21:44:11,396 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=e58a0dd97190346bf477f074c53b0411, ASSIGN in 399 msec 2023-07-11 21:44:11,399 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-11 21:44:11,400 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689111851400"}]},"ts":"1689111851400"} 2023-07-11 21:44:11,404 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-11 21:44:11,408 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-11 21:44:11,413 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 569 msec 2023-07-11 21:44:11,452 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-11 21:44:11,458 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-11 21:44:11,458 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 21:44:11,534 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-11 21:44:11,561 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-11 21:44:11,577 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 51 msec 2023-07-11 21:44:11,597 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-11 21:44:11,612 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-11 21:44:11,617 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 20 msec 2023-07-11 21:44:11,628 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-11 21:44:11,631 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-11 21:44:11,632 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.540sec 2023-07-11 21:44:11,635 INFO [master/jenkins-hbase5:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-11 21:44:11,637 INFO [master/jenkins-hbase5:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-11 21:44:11,637 INFO [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-11 21:44:11,638 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,43121,1689111846003-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-11 21:44:11,639 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,43121,1689111846003-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-11 21:44:11,647 DEBUG [Listener at localhost/45849] zookeeper.ReadOnlyZKClient(139): Connect 0x04ca72c7 to 127.0.0.1:60968 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 21:44:11,669 DEBUG [Listener at localhost/45849] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ca962b9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 21:44:11,677 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-11 21:44:11,696 DEBUG [hconnection-0x31531590-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 21:44:11,711 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:33070, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 21:44:11,724 INFO [Listener at localhost/45849] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase5.apache.org,43121,1689111846003 2023-07-11 21:44:11,736 DEBUG [Listener at localhost/45849] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-11 21:44:11,740 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:36622, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-11 21:44:11,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43121] master.HMaster$4(2112): Client=jenkins//172.31.10.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-11 21:44:11,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43121] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-11 21:44:11,762 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-11 21:44:11,764 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-11 21:44:11,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43121] master.MasterRpcServices(700): Client=jenkins//172.31.10.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-11 21:44:11,771 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/.tmp/data/default/t/a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:11,773 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/.tmp/data/default/t/a76d19c20500bab33c15f3dd1daa0f65 empty. 2023-07-11 21:44:11,775 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/.tmp/data/default/t/a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:11,776 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-11 21:44:11,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43121] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-11 21:44:11,805 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-11 21:44:11,807 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => a76d19c20500bab33c15f3dd1daa0f65, NAME => 't,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/.tmp 2023-07-11 21:44:11,857 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 21:44:11,857 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing a76d19c20500bab33c15f3dd1daa0f65, disabling compactions & flushes 2023-07-11 21:44:11,857 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:11,857 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:11,857 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. after waiting 0 ms 2023-07-11 21:44:11,857 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:11,857 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:11,857 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for a76d19c20500bab33c15f3dd1daa0f65: 2023-07-11 21:44:11,862 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-11 21:44:11,864 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689111851864"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689111851864"}]},"ts":"1689111851864"} 2023-07-11 21:44:11,867 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-11 21:44:11,869 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-11 21:44:11,869 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689111851869"}]},"ts":"1689111851869"} 2023-07-11 21:44:11,872 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-11 21:44:11,875 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-11 21:44:11,877 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-11 21:44:11,877 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-11 21:44:11,877 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-11 21:44:11,877 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=a76d19c20500bab33c15f3dd1daa0f65, ASSIGN}] 2023-07-11 21:44:11,879 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=a76d19c20500bab33c15f3dd1daa0f65, ASSIGN 2023-07-11 21:44:11,881 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=a76d19c20500bab33c15f3dd1daa0f65, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,43079,1689111847846; forceNewPlan=false, retain=false 2023-07-11 21:44:11,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43121] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-11 21:44:12,031 INFO [jenkins-hbase5:43121] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-11 21:44:12,032 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=a76d19c20500bab33c15f3dd1daa0f65, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,43079,1689111847846 2023-07-11 21:44:12,033 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689111852032"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689111852032"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689111852032"}]},"ts":"1689111852032"} 2023-07-11 21:44:12,037 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure a76d19c20500bab33c15f3dd1daa0f65, server=jenkins-hbase5.apache.org,43079,1689111847846}] 2023-07-11 21:44:12,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43121] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-11 21:44:12,199 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase5.apache.org,43079,1689111847846 2023-07-11 21:44:12,200 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-11 21:44:12,205 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:40994, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-11 21:44:12,212 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:12,212 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a76d19c20500bab33c15f3dd1daa0f65, NAME => 't,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.', STARTKEY => '', ENDKEY => ''} 2023-07-11 21:44:12,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:12,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 21:44:12,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:12,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:12,227 INFO [StoreOpener-a76d19c20500bab33c15f3dd1daa0f65-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:12,231 DEBUG [StoreOpener-a76d19c20500bab33c15f3dd1daa0f65-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/f 2023-07-11 21:44:12,231 DEBUG [StoreOpener-a76d19c20500bab33c15f3dd1daa0f65-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/f 2023-07-11 21:44:12,232 INFO [StoreOpener-a76d19c20500bab33c15f3dd1daa0f65-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a76d19c20500bab33c15f3dd1daa0f65 columnFamilyName f 2023-07-11 21:44:12,237 INFO [StoreOpener-a76d19c20500bab33c15f3dd1daa0f65-1] regionserver.HStore(310): Store=a76d19c20500bab33c15f3dd1daa0f65/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 21:44:12,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:12,249 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:12,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:12,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 21:44:12,283 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened a76d19c20500bab33c15f3dd1daa0f65; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67966613, jitterRate=0.012781456112861633}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 21:44:12,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for a76d19c20500bab33c15f3dd1daa0f65: 2023-07-11 21:44:12,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65., pid=11, masterSystemTime=1689111852199 2023-07-11 21:44:12,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:12,293 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:12,293 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=a76d19c20500bab33c15f3dd1daa0f65, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,43079,1689111847846 2023-07-11 21:44:12,294 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689111852293"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689111852293"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689111852293"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689111852293"}]},"ts":"1689111852293"} 2023-07-11 21:44:12,301 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-11 21:44:12,301 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure a76d19c20500bab33c15f3dd1daa0f65, server=jenkins-hbase5.apache.org,43079,1689111847846 in 260 msec 2023-07-11 21:44:12,305 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-11 21:44:12,305 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=a76d19c20500bab33c15f3dd1daa0f65, ASSIGN in 424 msec 2023-07-11 21:44:12,306 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-11 21:44:12,307 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689111852307"}]},"ts":"1689111852307"} 2023-07-11 21:44:12,310 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-11 21:44:12,317 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-11 21:44:12,321 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 561 msec 2023-07-11 21:44:12,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43121] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-11 21:44:12,392 INFO [Listener at localhost/45849] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-11 21:44:12,392 DEBUG [Listener at localhost/45849] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-11 21:44:12,394 INFO [Listener at localhost/45849] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-11 21:44:12,419 INFO [Listener at localhost/45849] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-11 21:44:12,419 INFO [Listener at localhost/45849] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-11 21:44:12,419 INFO [Listener at localhost/45849] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-11 21:44:12,438 DEBUG [Listener at localhost/45849] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 21:44:12,446 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:40996, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 21:44:12,516 INFO [Listener at localhost/45849] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=293, OpenFileDescriptor=585, MaxFileDescriptor=60000, SystemLoadAverage=610, ProcessCount=175, AvailableMemoryMB=3826 2023-07-11 21:44:12,518 INFO [Listener at localhost/45849] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-11 21:44:12,540 DEBUG [Listener at localhost/45849] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-11 21:44:12,541 DEBUG [Listener at localhost/45849] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:12,541 INFO [Listener at localhost/45849] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-11 21:44:12,541 INFO [Listener at localhost/45849] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-11 21:44:12,554 INFO [Listener at localhost/45849] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689111851400/Put/vlen=2/seqid=0} 2023-07-11 21:44:12,555 INFO [Listener at localhost/45849] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-11 21:44:12,555 INFO [Listener at localhost/45849] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411./info:regioninfo/1689111851380/Put/vlen=41/seqid=0, hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411./info:seqnumDuringOpen/1689111851380/Put/vlen=8/seqid=0, hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411./info:server/1689111851380/Put/vlen=31/seqid=0, hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411./info:serverstartcode/1689111851380/Put/vlen=8/seqid=0, hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411./info:sn/1689111851154/Put/vlen=45/seqid=0, hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411./info:state/1689111851380/Put/vlen=4/seqid=0} 2023-07-11 21:44:12,555 INFO [Listener at localhost/45849] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => e58a0dd97190346bf477f074c53b0411, NAME => 'hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411.', STARTKEY => '', ENDKEY => ''} 2023-07-11 21:44:12,556 INFO [Listener at localhost/45849] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689111852307/Put/vlen=2/seqid=0} 2023-07-11 21:44:12,556 INFO [Listener at localhost/45849] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-11 21:44:12,556 INFO [Listener at localhost/45849] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65./info:regioninfo/1689111852293/Put/vlen=35/seqid=0, t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65./info:seqnumDuringOpen/1689111852293/Put/vlen=8/seqid=0, t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65./info:server/1689111852293/Put/vlen=31/seqid=0, t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65./info:serverstartcode/1689111852293/Put/vlen=8/seqid=0, t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65./info:sn/1689111852032/Put/vlen=45/seqid=0, t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65./info:state/1689111852293/Put/vlen=4/seqid=0} 2023-07-11 21:44:12,557 INFO [Listener at localhost/45849] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => a76d19c20500bab33c15f3dd1daa0f65, NAME => 't,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.', STARTKEY => '', ENDKEY => ''} 2023-07-11 21:44:12,558 DEBUG [Listener at localhost/45849] zookeeper.ReadOnlyZKClient(139): Connect 0x1e8cbdf7 to 127.0.0.1:60968 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 21:44:12,588 DEBUG [Listener at localhost/45849] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@420100ae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 21:44:12,588 INFO [Listener at localhost/45849] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-11 21:44:12,588 INFO [Listener at localhost/45849] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-11 21:44:12,600 DEBUG [hconnection-0x31a3e955-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 21:44:12,607 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:33076, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 21:44:12,616 DEBUG [hconnection-0x31a3e955-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 21:44:12,621 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:41004, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 21:44:12,623 INFO [Listener at localhost/45849] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-11 21:44:12,627 ERROR [Listener at localhost/45849] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase5.apache.org,43079,1689111847846: die! ***** 2023-07-11 21:44:12,627 ERROR [Listener at localhost/45849] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-11 21:44:12,631 DEBUG [Listener at localhost/45849] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-11 21:44:12,635 DEBUG [Listener at localhost/45849] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-11 21:44:12,635 DEBUG [Listener at localhost/45849] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-11 21:44:12,636 DEBUG [Listener at localhost/45849] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-11 21:44:12,636 INFO [Listener at localhost/45849] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1166540800, "init": 513802240, "max": 2051014656, "used": 446641840 }, "NonHeapMemoryUsage": { "committed": 112615424, "init": 2555904, "max": -1, "used": 110503936 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase5.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase5.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase5.apache.org" } ] } 2023-07-11 21:44:12,641 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43121] master.MasterRpcServices(612): jenkins-hbase5.apache.org,43079,1689111847846 reported a fatal error: ***** ABORTING region server jenkins-hbase5.apache.org,43079,1689111847846: die! ***** 2023-07-11 21:44:12,647 INFO [Listener at localhost/45849] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,43079,1689111847846' ***** 2023-07-11 21:44:12,648 INFO [Listener at localhost/45849] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-11 21:44:12,649 INFO [RS:0;jenkins-hbase5:43079] regionserver.HeapMemoryManager(220): Stopping 2023-07-11 21:44:12,649 INFO [RS:0;jenkins-hbase5:43079] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-11 21:44:12,649 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-11 21:44:12,649 INFO [RS:0;jenkins-hbase5:43079] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-11 21:44:12,649 INFO [RS:0;jenkins-hbase5:43079] regionserver.HRegionServer(3305): Received CLOSE for a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:12,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43079] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner -8291362540189908991 2023-07-11 21:44:12,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43079] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase5.apache.org,43079,1689111847846 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 21:44:12,651 INFO [RS:0;jenkins-hbase5:43079] regionserver.HRegionServer(1141): aborting server jenkins-hbase5.apache.org,43079,1689111847846 2023-07-11 21:44:12,652 DEBUG [RS:0;jenkins-hbase5:43079] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0c21f3da to 127.0.0.1:60968 2023-07-11 21:44:12,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43079] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 172.31.10.131:41004 deadline: 1689111912651, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase5.apache.org,43079,1689111847846 aborting 2023-07-11 21:44:12,652 DEBUG [RS:0;jenkins-hbase5:43079] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 21:44:12,652 INFO [RS:0;jenkins-hbase5:43079] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-11 21:44:12,652 DEBUG [RS:0;jenkins-hbase5:43079] regionserver.HRegionServer(1478): Online Regions={a76d19c20500bab33c15f3dd1daa0f65=t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.} 2023-07-11 21:44:12,653 DEBUG [RS:0;jenkins-hbase5:43079] regionserver.HRegionServer(1504): Waiting on a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:12,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing a76d19c20500bab33c15f3dd1daa0f65, disabling compactions & flushes 2023-07-11 21:44:12,657 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:12,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:12,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. after waiting 0 ms 2023-07-11 21:44:12,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:12,664 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:12,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:12,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for a76d19c20500bab33c15f3dd1daa0f65: 2023-07-11 21:44:12,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:12,677 DEBUG [Listener at localhost/45849] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase5.apache.org,43079,1689111847846 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase5.apache.org,43079,1689111847846 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-11 21:44:12,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43079] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase5.apache.org,43079,1689111847846 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 21:44:12,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43079] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.10.131:41004 deadline: 1689111912680, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase5.apache.org,43079,1689111847846 aborting 2023-07-11 21:44:12,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43079] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase5.apache.org,43079,1689111847846 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 21:44:12,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43079] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Scan size: 125 connection: 172.31.10.131:41004 deadline: 1689111912791, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase5.apache.org,43079,1689111847846 aborting 2023-07-11 21:44:12,854 INFO [RS:0;jenkins-hbase5:43079] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,43079,1689111847846; all regions closed. 2023-07-11 21:44:12,877 DEBUG [RS:0;jenkins-hbase5:43079] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 21:44:12,887 INFO [RS:0;jenkins-hbase5:43079] regionserver.LeaseManager(133): Closed leases 2023-07-11 21:44:12,888 INFO [RS:0;jenkins-hbase5:43079] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase5:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-11 21:44:12,888 INFO [RS:0;jenkins-hbase5:43079] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-11 21:44:12,888 INFO [regionserver/jenkins-hbase5:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 21:44:12,888 INFO [RS:0;jenkins-hbase5:43079] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-11 21:44:12,889 INFO [RS:0;jenkins-hbase5:43079] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-11 21:44:12,890 INFO [RS:0;jenkins-hbase5:43079] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:43079 2023-07-11 21:44:12,906 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 21:44:12,906 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:37187-0x101c1f422730002, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase5.apache.org,43079,1689111847846 2023-07-11 21:44:12,906 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:37187-0x101c1f422730002, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 21:44:12,906 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:43079-0x101c1f422730001, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase5.apache.org,43079,1689111847846 2023-07-11 21:44:12,906 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:43079-0x101c1f422730001, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 21:44:12,913 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase5.apache.org,43079,1689111847846] 2023-07-11 21:44:12,913 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase5.apache.org,43079,1689111847846; numProcessing=1 2023-07-11 21:44:12,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37187-0x101c1f422730002, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:12,915 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase5.apache.org,43079,1689111847846 znode expired, triggering replicatorRemoved event 2023-07-11 21:44:12,916 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase5.apache.org,43079,1689111847846 already deleted, retry=false 2023-07-11 21:44:12,916 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase5.apache.org,43079,1689111847846 on jenkins-hbase5.apache.org,43121,1689111846003 2023-07-11 21:44:12,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37187-0x101c1f422730002, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:12,928 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase5.apache.org,43079,1689111847846, splitWal=true, meta=false 2023-07-11 21:44:12,928 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase5.apache.org,43079,1689111847846 (carryingMeta=false) jenkins-hbase5.apache.org,43079,1689111847846/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@7bc00b76[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-11 21:44:12,933 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase5.apache.org,43079,1689111847846, splitWal=true, meta=false 2023-07-11 21:44:12,935 INFO [PEWorker-1] procedure.ServerCrashProcedure(199): jenkins-hbase5.apache.org,43079,1689111847846 had 1 regions 2023-07-11 21:44:12,938 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase5.apache.org,43079,1689111847846, splitWal=true, meta=false, isMeta: false 2023-07-11 21:44:12,940 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846-splitting 2023-07-11 21:44:12,943 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase5.apache.org,43079,1689111847846 WAL count=1, meta=false 2023-07-11 21:44:12,946 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189}] 2023-07-11 21:44:12,957 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:12,959 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189, worker=jenkins-hbase5.apache.org,37187,1689111847969}] 2023-07-11 21:44:12,993 INFO [regionserver/jenkins-hbase5:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-11 21:44:13,003 WARN [RPCClient-NioEventLoopGroup-5-5] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase5.apache.org/172.31.10.131:43079 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:43079 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-11 21:44:13,005 DEBUG [RPCClient-NioEventLoopGroup-5-5] ipc.FailedServers(52): Added failed server with address jenkins-hbase5.apache.org/172.31.10.131:43079 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:43079 2023-07-11 21:44:13,014 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:43079-0x101c1f422730001, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 21:44:13,014 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:43079-0x101c1f422730001, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 21:44:13,015 INFO [RS:0;jenkins-hbase5:43079] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,43079,1689111847846; zookeeper connection closed. 2023-07-11 21:44:13,019 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3d8bbe68] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3d8bbe68 2023-07-11 21:44:13,129 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=37187] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-11 21:44:13,157 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846-splitting/jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189, size=1.1 K (1148bytes) 2023-07-11 21:44:13,158 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846-splitting/jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189 2023-07-11 21:44:13,164 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846-splitting/jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189 after 5ms 2023-07-11 21:44:13,171 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] wal.WALSplitter(311): Open hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846-splitting/jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189 took 15ms 2023-07-11 21:44:13,195 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] wal.WALSplitter(346): Last flushed sequenceid for a76d19c20500bab33c15f3dd1daa0f65: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-11 21:44:13,199 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846-splitting/jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189 so closing down 2023-07-11 21:44:13,199 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-11 21:44:13,213 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-11 21:44:13,216 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189.temp 2023-07-11 21:44:13,217 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-11 21:44:13,235 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-11 21:44:13,237 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189.temp to hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/recovered.edits/0000000000000000013 2023-07-11 21:44:13,238 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 59 ms; skipped=1; WAL=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846-splitting/jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189, size=1.1 K, length=1148, corrupted=false, cancelled=false 2023-07-11 21:44:13,238 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846-splitting/jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189, journal: Splitting hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846-splitting/jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189, size=1.1 K (1148bytes) at 1689111853158Finishing writing output for hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846-splitting/jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189 so closing down at 1689111853199 (+41 ms)Creating recovered edits writer path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189.temp at 1689111853216 (+17 ms)3 split writer threads finished at 1689111853217 (+1 ms)Closed recovered edits writer path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689111853235 (+18 ms)Rename recovered edits hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189.temp to hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/recovered.edits/0000000000000000013 at 1689111853237 (+2 ms)Processed 11 edits across 1 Regions in 59 ms; skipped=1; WAL=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846-splitting/jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189, size=1.1 K, length=1148, corrupted=false, cancelled=false at 1689111853238 (+1 ms) 2023-07-11 21:44:13,238 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846-splitting/jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189 2023-07-11 21:44:13,241 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-11 21:44:13,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43121] master.HMaster(3641): Remote procedure done, pid=14 2023-07-11 21:44:13,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43121] wal.WALSplitUtil(141): Moved hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846-splitting/jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189 to hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/oldWALs 2023-07-11 21:44:13,257 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-11 21:44:13,257 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189, worker=jenkins-hbase5.apache.org,37187,1689111847969 in 295 msec 2023-07-11 21:44:13,259 DEBUG [PEWorker-2] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:13,263 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-11 21:44:13,264 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase5.apache.org%2C43079%2C1689111847846.1689111850189, worker=jenkins-hbase5.apache.org,37187,1689111847969 in 316 msec 2023-07-11 21:44:13,266 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846-splitting dir is empty, no logs to split. 2023-07-11 21:44:13,266 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase5.apache.org,43079,1689111847846 WAL count=0, meta=false 2023-07-11 21:44:13,266 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase5.apache.org,43079,1689111847846 WAL splitting is done? wals=0, meta=false 2023-07-11 21:44:13,269 INFO [PEWorker-1] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase5.apache.org,43079,1689111847846 failed, ignore...File hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,43079,1689111847846-splitting does not exist. 2023-07-11 21:44:13,271 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=a76d19c20500bab33c15f3dd1daa0f65, ASSIGN}] 2023-07-11 21:44:13,273 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=a76d19c20500bab33c15f3dd1daa0f65, ASSIGN 2023-07-11 21:44:13,274 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=a76d19c20500bab33c15f3dd1daa0f65, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-11 21:44:13,312 DEBUG [hconnection-0x31a3e955-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase5.apache.org/172.31.10.131:43079 this server is in the failed servers list 2023-07-11 21:44:13,424 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=a76d19c20500bab33c15f3dd1daa0f65, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:13,425 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689111853424"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689111853424"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689111853424"}]},"ts":"1689111853424"} 2023-07-11 21:44:13,428 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure a76d19c20500bab33c15f3dd1daa0f65, server=jenkins-hbase5.apache.org,37187,1689111847969}] 2023-07-11 21:44:13,587 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:13,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a76d19c20500bab33c15f3dd1daa0f65, NAME => 't,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.', STARTKEY => '', ENDKEY => ''} 2023-07-11 21:44:13,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:13,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 21:44:13,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:13,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:13,590 INFO [StoreOpener-a76d19c20500bab33c15f3dd1daa0f65-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:13,591 DEBUG [StoreOpener-a76d19c20500bab33c15f3dd1daa0f65-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/f 2023-07-11 21:44:13,591 DEBUG [StoreOpener-a76d19c20500bab33c15f3dd1daa0f65-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/f 2023-07-11 21:44:13,591 INFO [StoreOpener-a76d19c20500bab33c15f3dd1daa0f65-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a76d19c20500bab33c15f3dd1daa0f65 columnFamilyName f 2023-07-11 21:44:13,592 INFO [StoreOpener-a76d19c20500bab33c15f3dd1daa0f65-1] regionserver.HStore(310): Store=a76d19c20500bab33c15f3dd1daa0f65/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 21:44:13,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:13,597 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:13,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/recovered.edits/0000000000000000013 2023-07-11 21:44:13,607 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/recovered.edits/0000000000000000013 2023-07-11 21:44:13,608 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing a76d19c20500bab33c15f3dd1daa0f65 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-11 21:44:13,699 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/.tmp/f/2c029f8ebc2748b8b80099cad006714e 2023-07-11 21:44:13,750 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/.tmp/f/2c029f8ebc2748b8b80099cad006714e as hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/f/2c029f8ebc2748b8b80099cad006714e 2023-07-11 21:44:13,763 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/f/2c029f8ebc2748b8b80099cad006714e, entries=10, sequenceid=13, filesize=4.8 K 2023-07-11 21:44:13,764 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for a76d19c20500bab33c15f3dd1daa0f65 in 156ms, sequenceid=13, compaction requested=false; wal=null 2023-07-11 21:44:13,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-11 21:44:13,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/recovered.edits/0000000000000000013 2023-07-11 21:44:13,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:13,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-11 21:44:13,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened a76d19c20500bab33c15f3dd1daa0f65; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62651324, jitterRate=-0.06642252206802368}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 21:44:13,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for a76d19c20500bab33c15f3dd1daa0f65: 2023-07-11 21:44:13,798 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65., pid=16, masterSystemTime=1689111853581 2023-07-11 21:44:13,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:13,801 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:13,802 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=a76d19c20500bab33c15f3dd1daa0f65, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:13,803 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689111853802"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689111853802"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689111853802"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689111853802"}]},"ts":"1689111853802"} 2023-07-11 21:44:13,810 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-11 21:44:13,810 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure a76d19c20500bab33c15f3dd1daa0f65, server=jenkins-hbase5.apache.org,37187,1689111847969 in 379 msec 2023-07-11 21:44:13,813 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-11 21:44:13,813 INFO [PEWorker-4] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase5.apache.org,43079,1689111847846 after splitting done 2023-07-11 21:44:13,813 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=a76d19c20500bab33c15f3dd1daa0f65, ASSIGN in 539 msec 2023-07-11 21:44:13,813 DEBUG [PEWorker-4] master.DeadServer(114): Removed jenkins-hbase5.apache.org,43079,1689111847846 from processing; numProcessing=0 2023-07-11 21:44:13,816 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase5.apache.org,43079,1689111847846, splitWal=true, meta=false in 894 msec 2023-07-11 21:44:13,843 DEBUG [Listener at localhost/45849] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e8cbdf7 to 127.0.0.1:60968 2023-07-11 21:44:13,843 DEBUG [Listener at localhost/45849] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 21:44:13,844 INFO [Listener at localhost/45849] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-11 21:44:13,862 INFO [Listener at localhost/45849] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=267 (was 293), OpenFileDescriptor=567 (was 585), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=610 (was 610), ProcessCount=175 (was 175), AvailableMemoryMB=3779 (was 3826) 2023-07-11 21:44:13,876 INFO [Listener at localhost/45849] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=267, OpenFileDescriptor=567, MaxFileDescriptor=60000, SystemLoadAverage=610, ProcessCount=175, AvailableMemoryMB=3778 2023-07-11 21:44:13,877 INFO [Listener at localhost/45849] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase5:43079 2023-07-11 21:44:13,891 INFO [Listener at localhost/45849] client.ConnectionUtils(127): regionserver/jenkins-hbase5:0 server-side Connection retries=45 2023-07-11 21:44:13,891 INFO [Listener at localhost/45849] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 21:44:13,892 INFO [Listener at localhost/45849] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 21:44:13,892 INFO [Listener at localhost/45849] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 21:44:13,892 INFO [Listener at localhost/45849] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 21:44:13,892 INFO [Listener at localhost/45849] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 21:44:13,892 INFO [Listener at localhost/45849] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 21:44:13,896 INFO [Listener at localhost/45849] ipc.NettyRpcServer(120): Bind to /172.31.10.131:36907 2023-07-11 21:44:13,897 INFO [Listener at localhost/45849] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-11 21:44:13,898 DEBUG [Listener at localhost/45849] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-11 21:44:13,899 INFO [Listener at localhost/45849] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 21:44:13,902 INFO [Listener at localhost/45849] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 21:44:13,903 INFO [Listener at localhost/45849] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36907 connecting to ZooKeeper ensemble=127.0.0.1:60968 2023-07-11 21:44:13,908 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:369070x0, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 21:44:13,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:36907-0x101c1f422730008 connected 2023-07-11 21:44:13,909 DEBUG [Listener at localhost/45849] zookeeper.ZKUtil(162): regionserver:36907-0x101c1f422730008, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 21:44:13,910 DEBUG [Listener at localhost/45849] zookeeper.ZKUtil(162): regionserver:36907-0x101c1f422730008, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-11 21:44:13,912 DEBUG [Listener at localhost/45849] zookeeper.ZKUtil(164): regionserver:36907-0x101c1f422730008, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 21:44:13,912 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36907 2023-07-11 21:44:13,915 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36907 2023-07-11 21:44:13,915 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36907 2023-07-11 21:44:13,916 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36907 2023-07-11 21:44:13,916 DEBUG [Listener at localhost/45849] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36907 2023-07-11 21:44:13,919 INFO [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(951): ClusterId : 5ef11920-c85b-4d5d-931f-0e6db259ed46 2023-07-11 21:44:13,921 DEBUG [RS:2;jenkins-hbase5:36907] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-11 21:44:13,923 DEBUG [RS:2;jenkins-hbase5:36907] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-11 21:44:13,923 DEBUG [RS:2;jenkins-hbase5:36907] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-11 21:44:13,926 DEBUG [RS:2;jenkins-hbase5:36907] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-11 21:44:13,928 DEBUG [RS:2;jenkins-hbase5:36907] zookeeper.ReadOnlyZKClient(139): Connect 0x266f6c21 to 127.0.0.1:60968 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 21:44:13,932 DEBUG [RS:2;jenkins-hbase5:36907] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28a8869c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 21:44:13,932 DEBUG [RS:2;jenkins-hbase5:36907] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c970751, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-11 21:44:13,944 DEBUG [RS:2;jenkins-hbase5:36907] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase5:36907 2023-07-11 21:44:13,944 INFO [RS:2;jenkins-hbase5:36907] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-11 21:44:13,944 INFO [RS:2;jenkins-hbase5:36907] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-11 21:44:13,944 DEBUG [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(1022): About to register with Master. 2023-07-11 21:44:13,945 INFO [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase5.apache.org,43121,1689111846003 with isa=jenkins-hbase5.apache.org/172.31.10.131:36907, startcode=1689111853890 2023-07-11 21:44:13,945 DEBUG [RS:2;jenkins-hbase5:36907] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-11 21:44:13,949 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:54403, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-11 21:44:13,950 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=43121] master.ServerManager(394): Registering regionserver=jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:13,950 DEBUG [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d 2023-07-11 21:44:13,950 DEBUG [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:40431 2023-07-11 21:44:13,951 DEBUG [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-11 21:44:13,959 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 21:44:13,960 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:37187-0x101c1f422730002, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 21:44:13,960 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase5.apache.org,36907,1689111853890] 2023-07-11 21:44:13,961 DEBUG [RS:2;jenkins-hbase5:36907] zookeeper.ZKUtil(162): regionserver:36907-0x101c1f422730008, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:13,961 WARN [RS:2;jenkins-hbase5:36907] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 21:44:13,961 INFO [RS:2;jenkins-hbase5:36907] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 21:44:13,961 DEBUG [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(1948): logDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:13,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37187-0x101c1f422730002, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:13,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:37187-0x101c1f422730002, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:13,976 DEBUG [RS:2;jenkins-hbase5:36907] zookeeper.ZKUtil(162): regionserver:36907-0x101c1f422730008, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:13,977 DEBUG [RS:2;jenkins-hbase5:36907] zookeeper.ZKUtil(162): regionserver:36907-0x101c1f422730008, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:13,978 DEBUG [RS:2;jenkins-hbase5:36907] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-11 21:44:13,978 INFO [RS:2;jenkins-hbase5:36907] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-11 21:44:13,981 INFO [RS:2;jenkins-hbase5:36907] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-11 21:44:13,982 INFO [RS:2;jenkins-hbase5:36907] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-11 21:44:13,982 INFO [RS:2;jenkins-hbase5:36907] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:13,982 INFO [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-11 21:44:13,988 INFO [RS:2;jenkins-hbase5:36907] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:13,988 DEBUG [RS:2;jenkins-hbase5:36907] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:13,989 DEBUG [RS:2;jenkins-hbase5:36907] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:13,989 DEBUG [RS:2;jenkins-hbase5:36907] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:13,989 DEBUG [RS:2;jenkins-hbase5:36907] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:13,989 DEBUG [RS:2;jenkins-hbase5:36907] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:13,989 DEBUG [RS:2;jenkins-hbase5:36907] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-11 21:44:13,989 DEBUG [RS:2;jenkins-hbase5:36907] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:13,989 DEBUG [RS:2;jenkins-hbase5:36907] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:13,989 DEBUG [RS:2;jenkins-hbase5:36907] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:13,990 DEBUG [RS:2;jenkins-hbase5:36907] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-11 21:44:13,998 INFO [RS:2;jenkins-hbase5:36907] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:13,998 INFO [RS:2;jenkins-hbase5:36907] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:13,998 INFO [RS:2;jenkins-hbase5:36907] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:14,014 INFO [RS:2;jenkins-hbase5:36907] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-11 21:44:14,014 INFO [RS:2;jenkins-hbase5:36907] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,36907,1689111853890-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 21:44:14,030 INFO [RS:2;jenkins-hbase5:36907] regionserver.Replication(203): jenkins-hbase5.apache.org,36907,1689111853890 started 2023-07-11 21:44:14,030 INFO [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(1637): Serving as jenkins-hbase5.apache.org,36907,1689111853890, RpcServer on jenkins-hbase5.apache.org/172.31.10.131:36907, sessionid=0x101c1f422730008 2023-07-11 21:44:14,030 INFO [Listener at localhost/45849] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase5:36907,5,FailOnTimeoutGroup] 2023-07-11 21:44:14,030 DEBUG [RS:2;jenkins-hbase5:36907] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-11 21:44:14,030 INFO [Listener at localhost/45849] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase5.apache.org,43079,1689111847846 2023-07-11 21:44:14,031 DEBUG [RS:2;jenkins-hbase5:36907] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:14,031 INFO [Listener at localhost/45849] client.TestScannerTimeout(177): START ************ test3686b 2023-07-11 21:44:14,031 DEBUG [RS:2;jenkins-hbase5:36907] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,36907,1689111853890' 2023-07-11 21:44:14,031 DEBUG [RS:2;jenkins-hbase5:36907] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-11 21:44:14,032 DEBUG [RS:2;jenkins-hbase5:36907] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-11 21:44:14,035 DEBUG [RS:2;jenkins-hbase5:36907] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-11 21:44:14,035 DEBUG [RS:2;jenkins-hbase5:36907] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-11 21:44:14,035 DEBUG [RS:2;jenkins-hbase5:36907] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:14,035 DEBUG [RS:2;jenkins-hbase5:36907] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,36907,1689111853890' 2023-07-11 21:44:14,035 DEBUG [RS:2;jenkins-hbase5:36907] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-11 21:44:14,036 DEBUG [RS:2;jenkins-hbase5:36907] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-11 21:44:14,036 DEBUG [RS:2;jenkins-hbase5:36907] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-11 21:44:14,036 INFO [RS:2;jenkins-hbase5:36907] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-11 21:44:14,037 INFO [RS:2;jenkins-hbase5:36907] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-11 21:44:14,044 DEBUG [Listener at localhost/45849] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-11 21:44:14,044 DEBUG [Listener at localhost/45849] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:14,045 DEBUG [Listener at localhost/45849] zookeeper.ReadOnlyZKClient(139): Connect 0x44c30efb to 127.0.0.1:60968 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 21:44:14,052 DEBUG [Listener at localhost/45849] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37cccf27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 21:44:14,055 DEBUG [hconnection-0x1748fbbb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 21:44:14,059 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:33078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 21:44:14,140 INFO [RS:2;jenkins-hbase5:36907] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C36907%2C1689111853890, suffix=, logDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,36907,1689111853890, archiveDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/oldWALs, maxLogs=32 2023-07-11 21:44:14,169 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46851,DS-2a3066f7-c3e4-4af9-ae4c-b59ab55b4085,DISK] 2023-07-11 21:44:14,171 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43955,DS-f2c706a2-5200-4d57-a191-b35285403eb2,DISK] 2023-07-11 21:44:14,177 INFO [RS:2;jenkins-hbase5:36907] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,36907,1689111853890/jenkins-hbase5.apache.org%2C36907%2C1689111853890.1689111854141 2023-07-11 21:44:14,183 DEBUG [RS:2;jenkins-hbase5:36907] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46851,DS-2a3066f7-c3e4-4af9-ae4c-b59ab55b4085,DISK], DatanodeInfoWithStorage[127.0.0.1:43955,DS-f2c706a2-5200-4d57-a191-b35285403eb2,DISK]] 2023-07-11 21:44:16,100 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-11 21:44:16,170 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-11 21:44:16,171 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-11 21:44:16,206 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-11 21:44:16,207 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-11 21:44:29,098 INFO [regionserver/jenkins-hbase5:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease -117763965987061741 expired clientIPAndPort=172.31.10.131:33078, userName=jenkins, regionInfo=t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:31,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37187] regionserver.RSRpcServices(3094): Client tried to access missing scanner -117763965987061741 2023-07-11 21:44:31,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37187] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 172.31.10.131:33078 deadline: 1689111931071, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-117763965987061741'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-11 21:44:31,073 DEBUG [Listener at localhost/45849] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-117763965987061741'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-117763965987061741'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-117763965987061741'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-11 21:44:31,078 DEBUG [Listener at localhost/45849] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44c30efb to 127.0.0.1:60968 2023-07-11 21:44:31,079 DEBUG [Listener at localhost/45849] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 21:44:31,079 INFO [Listener at localhost/45849] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-11 21:44:31,095 INFO [Listener at localhost/45849] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=293 (was 267) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36907 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS:2;jenkins-hbase5:36907-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-671253426-172.31.10.131-1689111841947:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60968@0x266f6c21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1318590247.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36907 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36907 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2023113051_17 at /127.0.0.1:45928 [Receiving block BP-671253426-172.31.10.131-1689111841947:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=36907 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: jenkins-hbase5:36907Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d-prefix:jenkins-hbase5.apache.org,36907,1689111853890 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-671253426-172.31.10.131-1689111841947:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60968@0x44c30efb-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60968@0x44c30efb-SendThread(127.0.0.1:60968) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60968@0x266f6c21-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/cluster_fb640830-b4cd-c2e5-ade2-4bb61d87c207/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=36907 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=36907 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 java.lang.Thread.currentThread(Native Method) org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:840) org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:836) org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.access$700(DefaultChannelPipeline.java:46) org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelUnregistered(DefaultChannelPipeline.java:1392) org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:215) org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:195) org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:821) org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:821) org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60968@0x44c30efb java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1318590247.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase5:36907 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60968@0x266f6c21-SendThread(127.0.0.1:60968) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:40431 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36907 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2023113051_17 at /127.0.0.1:38572 [Receiving block BP-671253426-172.31.10.131-1689111841947:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=36907 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36907 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/cluster_fb640830-b4cd-c2e5-ade2-4bb61d87c207/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36907 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) - Thread LEAK? -, OpenFileDescriptor=571 (was 567) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=683 (was 610) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 175), AvailableMemoryMB=3389 (was 3778) 2023-07-11 21:44:31,114 INFO [Listener at localhost/45849] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=293, OpenFileDescriptor=571, MaxFileDescriptor=60000, SystemLoadAverage=683, ProcessCount=173, AvailableMemoryMB=3386 2023-07-11 21:44:31,115 INFO [Listener at localhost/45849] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase5:43079 2023-07-11 21:44:31,116 INFO [Listener at localhost/45849] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase5.apache.org,43079,1689111847846 2023-07-11 21:44:31,116 INFO [Listener at localhost/45849] client.TestScannerTimeout(104): START************ test2772 2023-07-11 21:44:31,119 DEBUG [Listener at localhost/45849] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-11 21:44:31,119 DEBUG [Listener at localhost/45849] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:31,120 DEBUG [Listener at localhost/45849] zookeeper.ReadOnlyZKClient(139): Connect 0x003d8b32 to 127.0.0.1:60968 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 21:44:31,127 DEBUG [Listener at localhost/45849] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d6e31d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 21:44:31,127 ERROR [Listener at localhost/45849] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase5.apache.org,37187,1689111847969: die! ***** 2023-07-11 21:44:31,128 ERROR [Listener at localhost/45849] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-11 21:44:31,128 DEBUG [Listener at localhost/45849] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-11 21:44:31,129 DEBUG [Listener at localhost/45849] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-11 21:44:31,135 DEBUG [Listener at localhost/45849] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-11 21:44:31,138 DEBUG [Listener at localhost/45849] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-11 21:44:31,161 INFO [Listener at localhost/45849] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1164967936, "init": 513802240, "max": 2051014656, "used": 178362824 }, "NonHeapMemoryUsage": { "committed": 122183680, "init": 2555904, "max": -1, "used": 119702456 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase5.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 11958, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 61, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 61, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 61, "TotalCallTime_min": 1, "TotalCallTime_max": 1, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 61, "ResponseSize_min": 197, "ResponseSize_max": 679, "ResponseSize_mean": 438, "ResponseSize_25th_percentile": 199, "ResponseSize_median": 202, "ResponseSize_75th_percentile": 676, "ResponseSize_90th_percentile": 678, "ResponseSize_95th_percentile": 678, "ResponseSize_98th_percentile": 678, "ResponseSize_99th_percentile": 678, "ResponseSize_99.9th_percentile": 678, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 4, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 61, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16683 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase5.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:60968", "tag.serverName": "jenkins-hbase5.apache.org,37187,1689111847969", "tag.clusterId": "5ef11920-c85b-4d5d-931f-0e6db259ed46", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase5.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 1, "maxStoreFileCount": 1, "memStoreSize": 3306, "storeFileSize": 4926, "maxStoreFileAge": 16250, "minStoreFileAge": 16250, "avgStoreFileAge": 16250, "numReferenceFiles": 0, "regionServerStartTime": 1689111847969, "averageRegionSize": 2744, "storeFileIndexSize": 320, "staticIndexSize": 31, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 546, "localBytesRead": 546, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803016, "blockCacheCount": 1, "blockCacheDataBlockCount": 1, "blockCacheSize": 602872, "blockCacheCountHitPercent": 66.66666666666666, "blockCacheExpressHitPercent": 66.66666666666666, "l1CacheSize": 602872, "l1CacheFreeSize": 819803016, "l1CacheCount": 1, "l1CacheEvictionCount": 0, "l1CacheHitCount": 2, "l1CacheMissCount": 1, "l1CacheHitRatio": 0.6666666666666666, "l1CacheMissRatio": 0.3333333333333333, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 1640, "ByteBuffAllocatorPoolAllocationBytes": 1863680, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 46, "totalRowActionRequestCount": 57, "readRequestCount": 43, "filteredReadRequestCount": 0, "writeRequestCount": 14, "rpcGetRequestCount": 10, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 22, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 14, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 2, "blockCacheHitCountPrimary": 2, "blockCacheHitCachingCount": 2, "blockCacheMissCount": 1, "blockCacheMissCountPrimary": 1, "blockCacheMissCachingCount": 1, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 1, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 2, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 10, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 270, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 21, "ScanTime_min": 1, "ScanTime_max": 1, "ScanTime_mean": 1, "ScanTime_25th_percentile": 1, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 14, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 1, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 10, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 4926, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 1, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 21, "ScanSize_min": 155, "ScanSize_max": 653, "ScanSize_mean": 404, "ScanSize_25th_percentile": 156, "ScanSize_median": 157, "ScanSize_75th_percentile": 652, "ScanSize_90th_percentile": 652, "ScanSize_95th_percentile": 652, "ScanSize_98th_percentile": 652, "ScanSize_99th_percentile": 652, "ScanSize_99.9th_percentile": 652, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 270, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 1, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.5488116480610756, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 24, "ServerWriteQueryPerSecond_mean_rate": 1.1328049436323673, "ServerWriteQueryPerSecond_1min_rate": 3.738243758742744, "ServerWriteQueryPerSecond_5min_rate": 4.565901237603426, "ServerWriteQueryPerSecond_15min_rate": 4.720662978343764 } ] } 2023-07-11 21:44:31,163 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43121] master.MasterRpcServices(612): jenkins-hbase5.apache.org,37187,1689111847969 reported a fatal error: ***** ABORTING region server jenkins-hbase5.apache.org,37187,1689111847969: die! ***** 2023-07-11 21:44:31,171 INFO [Listener at localhost/45849] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,37187,1689111847969' ***** 2023-07-11 21:44:31,173 INFO [Listener at localhost/45849] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-11 21:44:31,173 INFO [RS:1;jenkins-hbase5:37187] regionserver.HeapMemoryManager(220): Stopping 2023-07-11 21:44:31,173 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-11 21:44:31,173 INFO [RS:1;jenkins-hbase5:37187] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-11 21:44:31,174 INFO [RS:1;jenkins-hbase5:37187] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-11 21:44:31,174 INFO [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(3305): Received CLOSE for e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:31,176 INFO [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(3305): Received CLOSE for a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:31,177 INFO [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(1141): aborting server jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:31,177 DEBUG [RS:1;jenkins-hbase5:37187] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x359d0571 to 127.0.0.1:60968 2023-07-11 21:44:31,177 DEBUG [RS:1;jenkins-hbase5:37187] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 21:44:31,177 INFO [RS:1;jenkins-hbase5:37187] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-11 21:44:31,177 INFO [RS:1;jenkins-hbase5:37187] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-11 21:44:31,177 INFO [RS:1;jenkins-hbase5:37187] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-11 21:44:31,177 INFO [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-11 21:44:31,178 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing e58a0dd97190346bf477f074c53b0411, disabling compactions & flushes 2023-07-11 21:44:31,178 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:31,178 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:31,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. after waiting 1 ms 2023-07-11 21:44:31,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:31,180 INFO [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-11 21:44:31,181 DEBUG [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(1478): Online Regions={e58a0dd97190346bf477f074c53b0411=hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411., 1588230740=hbase:meta,,1.1588230740, a76d19c20500bab33c15f3dd1daa0f65=t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.} 2023-07-11 21:44:31,181 DEBUG [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(1504): Waiting on 1588230740, a76d19c20500bab33c15f3dd1daa0f65, e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:31,183 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-11 21:44:31,183 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-11 21:44:31,184 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-11 21:44:31,184 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-11 21:44:31,184 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-11 21:44:31,199 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1825): Memstore data size is 3228 in region hbase:meta,,1.1588230740 2023-07-11 21:44:31,200 DEBUG [hconnection-0x1927b04e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 21:44:31,199 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:31,201 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-11 21:44:31,201 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:31,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for e58a0dd97190346bf477f074c53b0411: 2023-07-11 21:44:31,201 INFO [regionserver/jenkins-hbase5:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-11 21:44:31,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:31,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing a76d19c20500bab33c15f3dd1daa0f65, disabling compactions & flushes 2023-07-11 21:44:31,203 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:31,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:31,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. after waiting 0 ms 2023-07-11 21:44:31,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:31,204 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-11 21:44:31,204 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-11 21:44:31,204 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-11 21:44:31,212 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:31,212 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for a76d19c20500bab33c15f3dd1daa0f65: 2023-07-11 21:44:31,212 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:31,212 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:41988, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 21:44:31,213 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=37187] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase5.apache.org,37187,1689111847969 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 21:44:31,213 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=37187] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.10.131:41988 deadline: 1689111931213, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase5.apache.org,37187,1689111847969 aborting 2023-07-11 21:44:31,318 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=37187] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase5.apache.org,37187,1689111847969 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 21:44:31,319 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=37187] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.10.131:41988 deadline: 1689111931318, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase5.apache.org,37187,1689111847969 aborting 2023-07-11 21:44:31,381 INFO [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,37187,1689111847969; all regions closed. 2023-07-11 21:44:31,420 DEBUG [RS:1;jenkins-hbase5:37187] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 21:44:31,420 INFO [RS:1;jenkins-hbase5:37187] regionserver.LeaseManager(133): Closed leases 2023-07-11 21:44:31,421 INFO [RS:1;jenkins-hbase5:37187] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase5:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-11 21:44:31,422 INFO [regionserver/jenkins-hbase5:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 21:44:31,423 INFO [RS:1;jenkins-hbase5:37187] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:37187 2023-07-11 21:44:31,442 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:37187-0x101c1f422730002, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:31,442 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:36907-0x101c1f422730008, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase5.apache.org,37187,1689111847969 2023-07-11 21:44:31,442 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 21:44:31,443 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:36907-0x101c1f422730008, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 21:44:31,450 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:37187-0x101c1f422730002, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 21:44:31,451 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase5.apache.org,37187,1689111847969] 2023-07-11 21:44:31,451 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase5.apache.org,37187,1689111847969; numProcessing=1 2023-07-11 21:44:31,455 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase5.apache.org,37187,1689111847969 already deleted, retry=false 2023-07-11 21:44:31,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36907-0x101c1f422730008, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:31,456 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase5.apache.org,37187,1689111847969 on jenkins-hbase5.apache.org,43121,1689111846003 2023-07-11 21:44:31,456 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase5.apache.org,37187,1689111847969 znode expired, triggering replicatorRemoved event 2023-07-11 21:44:31,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36907-0x101c1f422730008, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:31,458 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=17, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase5.apache.org,37187,1689111847969, splitWal=true, meta=true 2023-07-11 21:44:31,458 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=17 for jenkins-hbase5.apache.org,37187,1689111847969 (carryingMeta=true) jenkins-hbase5.apache.org,37187,1689111847969/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@7f309b8f[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-11 21:44:31,460 INFO [PEWorker-5] procedure.ServerCrashProcedure(161): Start pid=17, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase5.apache.org,37187,1689111847969, splitWal=true, meta=true 2023-07-11 21:44:31,462 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase5.apache.org,37187,1689111847969, splitWal=true, meta=true, isMeta: true 2023-07-11 21:44:31,467 DEBUG [PEWorker-5] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting 2023-07-11 21:44:31,470 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase5.apache.org,37187,1689111847969 WAL count=1, meta=true 2023-07-11 21:44:31,470 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta}] 2023-07-11 21:44:31,474 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:31,476 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta, worker=jenkins-hbase5.apache.org,36907,1689111853890}] 2023-07-11 21:44:31,524 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase5.apache.org/172.31.10.131:37187 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:37187 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-11 21:44:31,525 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase5.apache.org/172.31.10.131:37187 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:37187 2023-07-11 21:44:31,550 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:37187-0x101c1f422730002, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 21:44:31,551 INFO [RS:1;jenkins-hbase5:37187] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,37187,1689111847969; zookeeper connection closed. 2023-07-11 21:44:31,551 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:37187-0x101c1f422730002, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 21:44:31,551 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5ba69f18] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5ba69f18 2023-07-11 21:44:31,629 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:31,629 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-11 21:44:31,631 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:55480, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-11 21:44:31,631 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36907] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=19 2023-07-11 21:44:31,655 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta, size=4.0 K (4123bytes) 2023-07-11 21:44:31,655 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta 2023-07-11 21:44:31,656 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta after 1ms 2023-07-11 21:44:31,662 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] wal.WALSplitter(311): Open hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta took 7ms 2023-07-11 21:44:31,677 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-11 21:44:31,680 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta so closing down 2023-07-11 21:44:31,680 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-11 21:44:31,691 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta.temp 2023-07-11 21:44:31,691 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-11 21:44:31,709 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) 2023-07-11 21:44:31,712 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta.temp to hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-11 21:44:31,712 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] wal.WALSplitter(423): Processed 13 edits across 1 Regions in 42 ms; skipped=1; WAL=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta, size=4.0 K, length=4123, corrupted=false, cancelled=false 2023-07-11 21:44:31,713 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta, journal: Splitting hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta, size=4.0 K (4123bytes) at 1689111871655Finishing writing output for hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta so closing down at 1689111871680 (+25 ms)Creating recovered edits writer path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta.temp at 1689111871691 (+11 ms)3 split writer threads finished at 1689111871691Closed recovered edits writer path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) at 1689111871709 (+18 ms)Rename recovered edits hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta.temp to hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 at 1689111871712 (+3 ms)Processed 13 edits across 1 Regions in 42 ms; skipped=1; WAL=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta, size=4.0 K, length=4123, corrupted=false, cancelled=false at 1689111871712 2023-07-11 21:44:31,713 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta 2023-07-11 21:44:31,713 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=19 2023-07-11 21:44:31,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43121] master.HMaster(3641): Remote procedure done, pid=19 2023-07-11 21:44:31,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43121] wal.WALSplitUtil(141): Moved hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta to hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/oldWALs 2023-07-11 21:44:31,722 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=18 2023-07-11 21:44:31,722 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta, worker=jenkins-hbase5.apache.org,36907,1689111853890 in 243 msec 2023-07-11 21:44:31,725 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:31,730 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-11 21:44:31,730 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase5.apache.org%2C37187%2C1689111847969.meta.1689111850459.meta, worker=jenkins-hbase5.apache.org,36907,1689111853890 in 255 msec 2023-07-11 21:44:31,734 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting dir is empty, no logs to split. 2023-07-11 21:44:31,734 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase5.apache.org,37187,1689111847969 WAL count=0, meta=true 2023-07-11 21:44:31,734 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase5.apache.org,37187,1689111847969 WAL splitting is done? wals=0, meta=true 2023-07-11 21:44:31,736 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-11 21:44:31,738 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-11 21:44:31,739 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-11 21:44:31,828 DEBUG [hconnection-0x1927b04e-metaLookup-shared--pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase5.apache.org/172.31.10.131:37187 this server is in the failed servers list 2023-07-11 21:44:31,889 DEBUG [jenkins-hbase5:43121] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-11 21:44:31,890 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,36907,1689111853890, state=OPENING 2023-07-11 21:44:31,895 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-11 21:44:31,896 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 21:44:31,897 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,36907,1689111853890}] 2023-07-11 21:44:32,057 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-11 21:44:32,057 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 21:44:32,060 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C36907%2C1689111853890.meta, suffix=.meta, logDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,36907,1689111853890, archiveDir=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/oldWALs, maxLogs=32 2023-07-11 21:44:32,086 DEBUG [RS-EventLoopGroup-6-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43955,DS-f2c706a2-5200-4d57-a191-b35285403eb2,DISK] 2023-07-11 21:44:32,086 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46851,DS-2a3066f7-c3e4-4af9-ae4c-b59ab55b4085,DISK] 2023-07-11 21:44:32,091 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,36907,1689111853890/jenkins-hbase5.apache.org%2C36907%2C1689111853890.meta.1689111872061.meta 2023-07-11 21:44:32,107 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46851,DS-2a3066f7-c3e4-4af9-ae4c-b59ab55b4085,DISK], DatanodeInfoWithStorage[127.0.0.1:43955,DS-f2c706a2-5200-4d57-a191-b35285403eb2,DISK]] 2023-07-11 21:44:32,108 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-11 21:44:32,108 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-11 21:44:32,108 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-11 21:44:32,109 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-11 21:44:32,109 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-11 21:44:32,109 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 21:44:32,109 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-11 21:44:32,109 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-11 21:44:32,112 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-11 21:44:32,116 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/info 2023-07-11 21:44:32,117 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/info 2023-07-11 21:44:32,117 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-11 21:44:32,121 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 21:44:32,136 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-11 21:44:32,143 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/rep_barrier 2023-07-11 21:44:32,143 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/rep_barrier 2023-07-11 21:44:32,144 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-11 21:44:32,146 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 21:44:32,147 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-11 21:44:32,149 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/table 2023-07-11 21:44:32,149 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/table 2023-07-11 21:44:32,150 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-11 21:44:32,151 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 21:44:32,153 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740 2023-07-11 21:44:32,157 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740 2023-07-11 21:44:32,158 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-11 21:44:32,179 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5464): Applied 32, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=15, path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-11 21:44:32,180 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=3.15 KB heapSize=6.27 KB 2023-07-11 21:44:32,253 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.00 KB at sequenceid=15 (bloomFilter=false), to=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/.tmp/info/c18c7220335f4294b247dd8708284fa0 2023-07-11 21:44:32,299 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=15 (bloomFilter=false), to=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/.tmp/table/cbc186137cc640da894d065f5e3c12ef 2023-07-11 21:44:32,314 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/.tmp/info/c18c7220335f4294b247dd8708284fa0 as hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/info/c18c7220335f4294b247dd8708284fa0 2023-07-11 21:44:32,327 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/info/c18c7220335f4294b247dd8708284fa0, entries=28, sequenceid=15, filesize=7.7 K 2023-07-11 21:44:32,330 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/.tmp/table/cbc186137cc640da894d065f5e3c12ef as hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/table/cbc186137cc640da894d065f5e3c12ef 2023-07-11 21:44:32,332 WARN [ReadOnlyZKClient-127.0.0.1:60968@0x003d8b32] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-11 21:44:32,332 DEBUG [hconnection-0x1927b04e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 21:44:32,348 INFO [RS-EventLoopGroup-6-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:55482, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 21:44:32,349 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36907] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Scan size: 111 connection: 172.31.10.131:55482 deadline: 1689111932348, exception=org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region hbase:meta,,1 is opening on jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:32,352 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/table/cbc186137cc640da894d065f5e3c12ef, entries=4, sequenceid=15, filesize=4.7 K 2023-07-11 21:44:32,352 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.15 KB/3228, heapSize ~5.98 KB/6128, currentSize=0 B/0 for 1588230740 in 172ms, sequenceid=15, compaction requested=false; wal=null 2023-07-11 21:44:32,354 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-11 21:44:32,359 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-11 21:44:32,362 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-11 21:44:32,365 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=1 2023-07-11 21:44:32,366 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71797422, jitterRate=0.06986495852470398}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-11 21:44:32,366 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-11 21:44:32,368 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=21, masterSystemTime=1689111872050 2023-07-11 21:44:32,370 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-11 21:44:32,371 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-11 21:44:32,372 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,36907,1689111853890, state=OPEN 2023-07-11 21:44:32,374 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-11 21:44:32,374 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 21:44:32,378 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-11 21:44:32,378 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,36907,1689111853890 in 477 msec 2023-07-11 21:44:32,380 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=17 2023-07-11 21:44:32,380 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase5.apache.org,37187,1689111847969 had 3 regions 2023-07-11 21:44:32,380 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 642 msec 2023-07-11 21:44:32,381 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase5.apache.org,37187,1689111847969, splitWal=true, meta=true, isMeta: false 2023-07-11 21:44:32,384 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase5.apache.org,37187,1689111847969 WAL count=1, meta=false 2023-07-11 21:44:32,384 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181}] 2023-07-11 21:44:32,388 DEBUG [PEWorker-2] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:32,389 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181, worker=jenkins-hbase5.apache.org,36907,1689111853890}] 2023-07-11 21:44:32,542 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36907] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=23 2023-07-11 21:44:32,564 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181, size=899 (899bytes) 2023-07-11 21:44:32,564 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181 2023-07-11 21:44:32,565 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181 after 1ms 2023-07-11 21:44:32,580 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-1] wal.WALSplitter(311): Open hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181 took 17ms 2023-07-11 21:44:32,587 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-1] wal.WALSplitter(346): Last flushed sequenceid for e58a0dd97190346bf477f074c53b0411: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-11 21:44:32,590 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-1] wal.WALSplitter(346): Last flushed sequenceid for a76d19c20500bab33c15f3dd1daa0f65: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-11 21:44:32,590 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181 so closing down 2023-07-11 21:44:32,590 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-11 21:44:32,595 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181.temp 2023-07-11 21:44:32,596 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-11 21:44:32,612 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-11 21:44:32,614 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181.temp to hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/recovered.edits/0000000000000000005 2023-07-11 21:44:32,615 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-1] wal.WALSplitter(423): Processed 4 edits across 1 Regions in 31 ms; skipped=2; WAL=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181, size=899, length=899, corrupted=false, cancelled=false 2023-07-11 21:44:32,615 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181, journal: Splitting hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181, size=899 (899bytes) at 1689111872564Finishing writing output for hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181 so closing down at 1689111872590 (+26 ms)Creating recovered edits writer path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181.temp at 1689111872595 (+5 ms)3 split writer threads finished at 1689111872596 (+1 ms)Closed recovered edits writer path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689111872612 (+16 ms)Rename recovered edits hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/recovered.edits/0000000000000000004-jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181.temp to hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/recovered.edits/0000000000000000005 at 1689111872614 (+2 ms)Processed 4 edits across 1 Regions in 31 ms; skipped=2; WAL=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181, size=899, length=899, corrupted=false, cancelled=false at 1689111872615 (+1 ms) 2023-07-11 21:44:32,615 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181 2023-07-11 21:44:32,615 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=23 2023-07-11 21:44:32,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43121] master.HMaster(3641): Remote procedure done, pid=23 2023-07-11 21:44:32,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43121] wal.WALSplitUtil(141): Moved hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting/jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181 to hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/oldWALs 2023-07-11 21:44:32,625 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-11 21:44:32,625 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181, worker=jenkins-hbase5.apache.org,36907,1689111853890 in 232 msec 2023-07-11 21:44:32,628 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:32,631 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=17 2023-07-11 21:44:32,632 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase5.apache.org%2C37187%2C1689111847969.1689111850181, worker=jenkins-hbase5.apache.org,36907,1689111853890 in 244 msec 2023-07-11 21:44:32,634 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,37187,1689111847969-splitting dir is empty, no logs to split. 2023-07-11 21:44:32,634 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase5.apache.org,37187,1689111847969 WAL count=0, meta=false 2023-07-11 21:44:32,634 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase5.apache.org,37187,1689111847969 WAL splitting is done? wals=0, meta=false 2023-07-11 21:44:32,639 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=a76d19c20500bab33c15f3dd1daa0f65, ASSIGN}, {pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e58a0dd97190346bf477f074c53b0411, ASSIGN}] 2023-07-11 21:44:32,641 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e58a0dd97190346bf477f074c53b0411, ASSIGN 2023-07-11 21:44:32,641 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=a76d19c20500bab33c15f3dd1daa0f65, ASSIGN 2023-07-11 21:44:32,644 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=e58a0dd97190346bf477f074c53b0411, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-11 21:44:32,644 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=a76d19c20500bab33c15f3dd1daa0f65, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-11 21:44:32,794 DEBUG [jenkins-hbase5:43121] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-11 21:44:32,795 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e58a0dd97190346bf477f074c53b0411, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:32,795 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=a76d19c20500bab33c15f3dd1daa0f65, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:32,795 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689111872795"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689111872795"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689111872795"}]},"ts":"1689111872795"} 2023-07-11 21:44:32,795 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689111872795"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689111872795"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689111872795"}]},"ts":"1689111872795"} 2023-07-11 21:44:32,796 WARN [RS-EventLoopGroup-4-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase5.apache.org/172.31.10.131:37187 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:37187 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-11 21:44:32,798 DEBUG [RS-EventLoopGroup-4-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase5.apache.org/172.31.10.131:37187 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:37187 2023-07-11 21:44:32,900 DEBUG [PEWorker-3] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 21:44:32,901 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:55492, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 21:44:32,904 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=24, state=RUNNABLE; OpenRegionProcedure a76d19c20500bab33c15f3dd1daa0f65, server=jenkins-hbase5.apache.org,36907,1689111853890}] 2023-07-11 21:44:32,905 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=25, state=RUNNABLE; OpenRegionProcedure e58a0dd97190346bf477f074c53b0411, server=jenkins-hbase5.apache.org,36907,1689111853890}] 2023-07-11 21:44:33,061 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:33,061 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e58a0dd97190346bf477f074c53b0411, NAME => 'hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411.', STARTKEY => '', ENDKEY => ''} 2023-07-11 21:44:33,061 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:33,062 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 21:44:33,062 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:33,062 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:33,064 INFO [StoreOpener-e58a0dd97190346bf477f074c53b0411-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:33,066 DEBUG [StoreOpener-e58a0dd97190346bf477f074c53b0411-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/info 2023-07-11 21:44:33,066 DEBUG [StoreOpener-e58a0dd97190346bf477f074c53b0411-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/info 2023-07-11 21:44:33,067 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:33,067 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a76d19c20500bab33c15f3dd1daa0f65, NAME => 't,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.', STARTKEY => '', ENDKEY => ''} 2023-07-11 21:44:33,067 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:33,067 INFO [StoreOpener-e58a0dd97190346bf477f074c53b0411-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e58a0dd97190346bf477f074c53b0411 columnFamilyName info 2023-07-11 21:44:33,067 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 21:44:33,067 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:33,067 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:33,068 INFO [StoreOpener-e58a0dd97190346bf477f074c53b0411-1] regionserver.HStore(310): Store=e58a0dd97190346bf477f074c53b0411/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 21:44:33,069 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:33,070 INFO [StoreOpener-a76d19c20500bab33c15f3dd1daa0f65-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:33,071 DEBUG [StoreOpener-a76d19c20500bab33c15f3dd1daa0f65-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/f 2023-07-11 21:44:33,071 DEBUG [StoreOpener-a76d19c20500bab33c15f3dd1daa0f65-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/f 2023-07-11 21:44:33,074 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:33,074 INFO [StoreOpener-a76d19c20500bab33c15f3dd1daa0f65-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a76d19c20500bab33c15f3dd1daa0f65 columnFamilyName f 2023-07-11 21:44:33,075 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/recovered.edits/0000000000000000005 2023-07-11 21:44:33,080 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/recovered.edits/0000000000000000005 2023-07-11 21:44:33,081 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing e58a0dd97190346bf477f074c53b0411 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-11 21:44:33,086 DEBUG [StoreOpener-a76d19c20500bab33c15f3dd1daa0f65-1] regionserver.HStore(539): loaded hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/f/2c029f8ebc2748b8b80099cad006714e 2023-07-11 21:44:33,087 INFO [StoreOpener-a76d19c20500bab33c15f3dd1daa0f65-1] regionserver.HStore(310): Store=a76d19c20500bab33c15f3dd1daa0f65/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 21:44:33,088 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:33,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:33,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:33,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-11 21:44:33,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened a76d19c20500bab33c15f3dd1daa0f65; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63950809, jitterRate=-0.04705868661403656}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 21:44:33,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for a76d19c20500bab33c15f3dd1daa0f65: 2023-07-11 21:44:33,110 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65., pid=26, masterSystemTime=1689111873056 2023-07-11 21:44:33,115 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:33,115 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:33,121 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=a76d19c20500bab33c15f3dd1daa0f65, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:33,121 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689111873120"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689111873120"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689111873120"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689111873120"}]},"ts":"1689111873120"} 2023-07-11 21:44:33,123 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/.tmp/info/8d50fcdd7662463c92e5b8d2c5a1543d 2023-07-11 21:44:33,128 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=24 2023-07-11 21:44:33,128 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=24, state=SUCCESS; OpenRegionProcedure a76d19c20500bab33c15f3dd1daa0f65, server=jenkins-hbase5.apache.org,36907,1689111853890 in 221 msec 2023-07-11 21:44:33,130 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=t, region=a76d19c20500bab33c15f3dd1daa0f65, ASSIGN in 489 msec 2023-07-11 21:44:33,135 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/.tmp/info/8d50fcdd7662463c92e5b8d2c5a1543d as hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/info/8d50fcdd7662463c92e5b8d2c5a1543d 2023-07-11 21:44:33,145 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/info/8d50fcdd7662463c92e5b8d2c5a1543d, entries=2, sequenceid=5, filesize=4.8 K 2023-07-11 21:44:33,145 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for e58a0dd97190346bf477f074c53b0411 in 64ms, sequenceid=5, compaction requested=false; wal=null 2023-07-11 21:44:33,147 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/recovered.edits/0000000000000000005 2023-07-11 21:44:33,151 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:33,154 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-11 21:44:33,155 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened e58a0dd97190346bf477f074c53b0411; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73219982, jitterRate=0.0910627543926239}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 21:44:33,155 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for e58a0dd97190346bf477f074c53b0411: 2023-07-11 21:44:33,159 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411., pid=27, masterSystemTime=1689111873056 2023-07-11 21:44:33,162 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:33,162 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:33,163 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e58a0dd97190346bf477f074c53b0411, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:33,163 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689111873163"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689111873163"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689111873163"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689111873163"}]},"ts":"1689111873163"} 2023-07-11 21:44:33,169 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=25 2023-07-11 21:44:33,169 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=25, state=SUCCESS; OpenRegionProcedure e58a0dd97190346bf477f074c53b0411, server=jenkins-hbase5.apache.org,36907,1689111853890 in 261 msec 2023-07-11 21:44:33,172 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=17 2023-07-11 21:44:33,172 INFO [PEWorker-2] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase5.apache.org,37187,1689111847969 after splitting done 2023-07-11 21:44:33,172 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=e58a0dd97190346bf477f074c53b0411, ASSIGN in 530 msec 2023-07-11 21:44:33,172 DEBUG [PEWorker-2] master.DeadServer(114): Removed jenkins-hbase5.apache.org,37187,1689111847969 from processing; numProcessing=0 2023-07-11 21:44:33,174 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=17, state=SUCCESS; ServerCrashProcedure jenkins-hbase5.apache.org,37187,1689111847969, splitWal=true, meta=true in 1.7160 sec 2023-07-11 21:44:33,370 DEBUG [Listener at localhost/45849] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x003d8b32 to 127.0.0.1:60968 2023-07-11 21:44:33,370 DEBUG [Listener at localhost/45849] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 21:44:33,371 INFO [Listener at localhost/45849] client.TestScannerTimeout(124): END ************ test2772 2023-07-11 21:44:33,388 INFO [Listener at localhost/45849] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=274 (was 293), OpenFileDescriptor=581 (was 571) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=683 (was 683), ProcessCount=173 (was 173), AvailableMemoryMB=3324 (was 3386) 2023-07-11 21:44:33,389 INFO [Listener at localhost/45849] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-11 21:44:33,390 INFO [Listener at localhost/45849] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-11 21:44:33,390 DEBUG [Listener at localhost/45849] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04ca72c7 to 127.0.0.1:60968 2023-07-11 21:44:33,390 DEBUG [Listener at localhost/45849] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 21:44:33,390 DEBUG [Listener at localhost/45849] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-11 21:44:33,390 DEBUG [Listener at localhost/45849] util.JVMClusterUtil(257): Found active master hash=1576424157, stopped=false 2023-07-11 21:44:33,390 INFO [Listener at localhost/45849] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase5.apache.org,43121,1689111846003 2023-07-11 21:44:33,398 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-11 21:44:33,398 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:36907-0x101c1f422730008, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-11 21:44:33,398 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 21:44:33,398 INFO [Listener at localhost/45849] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 21:44:33,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 21:44:33,399 DEBUG [Listener at localhost/45849] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2e9dcc78 to 127.0.0.1:60968 2023-07-11 21:44:33,399 DEBUG [Listener at localhost/45849] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 21:44:33,399 INFO [Listener at localhost/45849] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,36907,1689111853890' ***** 2023-07-11 21:44:33,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36907-0x101c1f422730008, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 21:44:33,400 INFO [Listener at localhost/45849] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-11 21:44:33,400 INFO [RS:2;jenkins-hbase5:36907] regionserver.HeapMemoryManager(220): Stopping 2023-07-11 21:44:33,400 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-11 21:44:33,400 INFO [RS:2;jenkins-hbase5:36907] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-11 21:44:33,401 INFO [RS:2;jenkins-hbase5:36907] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-11 21:44:33,401 INFO [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(3305): Received CLOSE for e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:33,401 INFO [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(3305): Received CLOSE for a76d19c20500bab33c15f3dd1daa0f65 2023-07-11 21:44:33,401 INFO [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:33,402 DEBUG [RS:2;jenkins-hbase5:36907] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x266f6c21 to 127.0.0.1:60968 2023-07-11 21:44:33,402 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing e58a0dd97190346bf477f074c53b0411, disabling compactions & flushes 2023-07-11 21:44:33,402 DEBUG [RS:2;jenkins-hbase5:36907] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 21:44:33,402 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:33,402 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:33,402 INFO [RS:2;jenkins-hbase5:36907] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-11 21:44:33,402 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. after waiting 0 ms 2023-07-11 21:44:33,402 INFO [RS:2;jenkins-hbase5:36907] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-11 21:44:33,402 INFO [RS:2;jenkins-hbase5:36907] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-11 21:44:33,402 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:33,402 INFO [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-11 21:44:33,403 INFO [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-11 21:44:33,403 DEBUG [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(1478): Online Regions={e58a0dd97190346bf477f074c53b0411=hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411., 1588230740=hbase:meta,,1.1588230740, a76d19c20500bab33c15f3dd1daa0f65=t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65.} 2023-07-11 21:44:33,403 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-11 21:44:33,404 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-11 21:44:33,403 DEBUG [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(1504): Waiting on 1588230740, a76d19c20500bab33c15f3dd1daa0f65, e58a0dd97190346bf477f074c53b0411 2023-07-11 21:44:33,404 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-11 21:44:33,404 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-11 21:44:33,405 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-11 21:44:33,405 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-11 21:44:33,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/namespace/e58a0dd97190346bf477f074c53b0411/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=5 2023-07-11 21:44:33,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:33,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for e58a0dd97190346bf477f074c53b0411: 2023-07-11 21:44:33,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689111850838.e58a0dd97190346bf477f074c53b0411. 2023-07-11 21:44:33,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing a76d19c20500bab33c15f3dd1daa0f65, disabling compactions & flushes 2023-07-11 21:44:33,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:33,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:33,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. after waiting 0 ms 2023-07-11 21:44:33,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:33,436 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/default/t/a76d19c20500bab33c15f3dd1daa0f65/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-11 21:44:33,438 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:33,439 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for a76d19c20500bab33c15f3dd1daa0f65: 2023-07-11 21:44:33,439 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed t,,1689111851751.a76d19c20500bab33c15f3dd1daa0f65. 2023-07-11 21:44:33,443 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=22 (bloomFilter=false), to=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/.tmp/info/3b5713da408d4b0ebe73f41ced50ce43 2023-07-11 21:44:33,462 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/.tmp/info/3b5713da408d4b0ebe73f41ced50ce43 as hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/info/3b5713da408d4b0ebe73f41ced50ce43 2023-07-11 21:44:33,469 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/info/3b5713da408d4b0ebe73f41ced50ce43, entries=16, sequenceid=22, filesize=6.4 K 2023-07-11 21:44:33,470 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 65ms, sequenceid=22, compaction requested=false 2023-07-11 21:44:33,477 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/data/hbase/meta/1588230740/recovered.edits/25.seqid, newMaxSeqId=25, maxSeqId=15 2023-07-11 21:44:33,478 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-11 21:44:33,478 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-11 21:44:33,479 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-11 21:44:33,479 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-11 21:44:33,604 INFO [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,36907,1689111853890; all regions closed. 2023-07-11 21:44:33,609 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/WALs/jenkins-hbase5.apache.org,36907,1689111853890/jenkins-hbase5.apache.org%2C36907%2C1689111853890.meta.1689111872061.meta not finished, retry = 0 2023-07-11 21:44:33,713 DEBUG [RS:2;jenkins-hbase5:36907] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/oldWALs 2023-07-11 21:44:33,713 INFO [RS:2;jenkins-hbase5:36907] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C36907%2C1689111853890.meta:.meta(num 1689111872061) 2023-07-11 21:44:33,718 DEBUG [RS:2;jenkins-hbase5:36907] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/oldWALs 2023-07-11 21:44:33,718 INFO [RS:2;jenkins-hbase5:36907] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C36907%2C1689111853890:(num 1689111854141) 2023-07-11 21:44:33,719 DEBUG [RS:2;jenkins-hbase5:36907] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 21:44:33,719 INFO [RS:2;jenkins-hbase5:36907] regionserver.LeaseManager(133): Closed leases 2023-07-11 21:44:33,719 INFO [RS:2;jenkins-hbase5:36907] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase5:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-11 21:44:33,719 INFO [regionserver/jenkins-hbase5:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 21:44:33,720 INFO [RS:2;jenkins-hbase5:36907] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:36907 2023-07-11 21:44:33,722 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 21:44:33,722 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:36907-0x101c1f422730008, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase5.apache.org,36907,1689111853890 2023-07-11 21:44:33,722 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:36907-0x101c1f422730008, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 21:44:33,724 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase5.apache.org,36907,1689111853890] 2023-07-11 21:44:33,724 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase5.apache.org,36907,1689111853890; numProcessing=1 2023-07-11 21:44:33,726 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase5.apache.org,36907,1689111853890 already deleted, retry=false 2023-07-11 21:44:33,726 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase5.apache.org,36907,1689111853890 expired; onlineServers=0 2023-07-11 21:44:33,726 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,43121,1689111846003' ***** 2023-07-11 21:44:33,726 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-11 21:44:33,726 DEBUG [M:0;jenkins-hbase5:43121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ccb58e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-11 21:44:33,726 INFO [M:0;jenkins-hbase5:43121] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,43121,1689111846003 2023-07-11 21:44:33,726 INFO [M:0;jenkins-hbase5:43121] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,43121,1689111846003; all regions closed. 2023-07-11 21:44:33,726 DEBUG [M:0;jenkins-hbase5:43121] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 21:44:33,727 DEBUG [M:0;jenkins-hbase5:43121] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-11 21:44:33,727 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-11 21:44:33,727 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689111849758] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689111849758,5,FailOnTimeoutGroup] 2023-07-11 21:44:33,727 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689111849755] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689111849755,5,FailOnTimeoutGroup] 2023-07-11 21:44:33,727 DEBUG [M:0;jenkins-hbase5:43121] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-11 21:44:33,727 INFO [M:0;jenkins-hbase5:43121] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-11 21:44:33,727 INFO [M:0;jenkins-hbase5:43121] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-11 21:44:33,727 INFO [M:0;jenkins-hbase5:43121] hbase.ChoreService(369): Chore service for: master/jenkins-hbase5:0 had [] on shutdown 2023-07-11 21:44:33,728 DEBUG [M:0;jenkins-hbase5:43121] master.HMaster(1512): Stopping service threads 2023-07-11 21:44:33,728 INFO [M:0;jenkins-hbase5:43121] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-11 21:44:33,728 ERROR [M:0;jenkins-hbase5:43121] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (813912309) connection to localhost/127.0.0.1:40431 from jenkins,5,PEWorkerGroup] 2023-07-11 21:44:33,728 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 21:44:33,728 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 21:44:33,728 INFO [M:0;jenkins-hbase5:43121] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 21:44:33,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 21:44:33,729 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-11 21:44:33,729 DEBUG [M:0;jenkins-hbase5:43121] zookeeper.ZKUtil(398): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-11 21:44:33,729 WARN [M:0;jenkins-hbase5:43121] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-11 21:44:33,729 INFO [M:0;jenkins-hbase5:43121] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 21:44:33,729 INFO [M:0;jenkins-hbase5:43121] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-11 21:44:33,730 DEBUG [M:0;jenkins-hbase5:43121] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 21:44:33,730 INFO [M:0;jenkins-hbase5:43121] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 21:44:33,730 DEBUG [M:0;jenkins-hbase5:43121] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 21:44:33,730 DEBUG [M:0;jenkins-hbase5:43121] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 21:44:33,730 DEBUG [M:0;jenkins-hbase5:43121] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 21:44:33,730 INFO [M:0;jenkins-hbase5:43121] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=90.80 KB heapSize=109.53 KB 2023-07-11 21:44:33,750 INFO [M:0;jenkins-hbase5:43121] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=90.80 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e344c4c27a154e4eaee013fb2d28c6e3 2023-07-11 21:44:33,757 DEBUG [M:0;jenkins-hbase5:43121] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e344c4c27a154e4eaee013fb2d28c6e3 as hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e344c4c27a154e4eaee013fb2d28c6e3 2023-07-11 21:44:33,767 INFO [M:0;jenkins-hbase5:43121] regionserver.HStore(1080): Added hdfs://localhost:40431/user/jenkins/test-data/e0ee230b-992c-0206-bcc6-ed25c50c888d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e344c4c27a154e4eaee013fb2d28c6e3, entries=27, sequenceid=212, filesize=8.2 K 2023-07-11 21:44:33,768 INFO [M:0;jenkins-hbase5:43121] regionserver.HRegion(2948): Finished flush of dataSize ~90.80 KB/92977, heapSize ~109.52 KB/112144, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 38ms, sequenceid=212, compaction requested=false 2023-07-11 21:44:33,769 INFO [M:0;jenkins-hbase5:43121] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 21:44:33,770 DEBUG [M:0;jenkins-hbase5:43121] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 21:44:33,774 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 21:44:33,774 INFO [M:0;jenkins-hbase5:43121] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-11 21:44:33,777 INFO [M:0;jenkins-hbase5:43121] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:43121 2023-07-11 21:44:33,779 DEBUG [M:0;jenkins-hbase5:43121] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase5.apache.org,43121,1689111846003 already deleted, retry=false 2023-07-11 21:44:33,875 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:36907-0x101c1f422730008, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 21:44:33,875 INFO [RS:2;jenkins-hbase5:36907] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,36907,1689111853890; zookeeper connection closed. 2023-07-11 21:44:33,875 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): regionserver:36907-0x101c1f422730008, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 21:44:33,875 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7e6e7dd] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7e6e7dd 2023-07-11 21:44:33,876 INFO [Listener at localhost/45849] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-11 21:44:33,975 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 21:44:33,975 INFO [M:0;jenkins-hbase5:43121] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,43121,1689111846003; zookeeper connection closed. 2023-07-11 21:44:33,975 DEBUG [Listener at localhost/45849-EventThread] zookeeper.ZKWatcher(600): master:43121-0x101c1f422730000, quorum=127.0.0.1:60968, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 21:44:33,977 WARN [Listener at localhost/45849] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-11 21:44:33,982 INFO [Listener at localhost/45849] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-11 21:44:34,000 INFO [regionserver/jenkins-hbase5:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-11 21:44:34,088 WARN [BP-671253426-172.31.10.131-1689111841947 heartbeating to localhost/127.0.0.1:40431] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-11 21:44:34,088 WARN [BP-671253426-172.31.10.131-1689111841947 heartbeating to localhost/127.0.0.1:40431] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-671253426-172.31.10.131-1689111841947 (Datanode Uuid 937b8060-e4c8-4564-9ee6-99d9602b96e2) service to localhost/127.0.0.1:40431 2023-07-11 21:44:34,090 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/cluster_fb640830-b4cd-c2e5-ade2-4bb61d87c207/dfs/data/data3/current/BP-671253426-172.31.10.131-1689111841947] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-11 21:44:34,091 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/cluster_fb640830-b4cd-c2e5-ade2-4bb61d87c207/dfs/data/data4/current/BP-671253426-172.31.10.131-1689111841947] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-11 21:44:34,096 WARN [Listener at localhost/45849] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-11 21:44:34,101 INFO [Listener at localhost/45849] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-11 21:44:34,207 WARN [BP-671253426-172.31.10.131-1689111841947 heartbeating to localhost/127.0.0.1:40431] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-11 21:44:34,207 WARN [BP-671253426-172.31.10.131-1689111841947 heartbeating to localhost/127.0.0.1:40431] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-671253426-172.31.10.131-1689111841947 (Datanode Uuid 099977b4-8786-41f4-a006-ec41424f7611) service to localhost/127.0.0.1:40431 2023-07-11 21:44:34,208 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/cluster_fb640830-b4cd-c2e5-ade2-4bb61d87c207/dfs/data/data1/current/BP-671253426-172.31.10.131-1689111841947] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-11 21:44:34,208 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/26663ad4-7a10-c562-7aac-5052b5eead9a/cluster_fb640830-b4cd-c2e5-ade2-4bb61d87c207/dfs/data/data2/current/BP-671253426-172.31.10.131-1689111841947] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-11 21:44:34,249 INFO [Listener at localhost/45849] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-11 21:44:34,367 INFO [Listener at localhost/45849] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-11 21:44:34,424 INFO [Listener at localhost/45849] hbase.HBaseTestingUtility(1293): Minicluster is down