2023-07-13 06:42:18,164 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-13 06:42:18,376 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370 2023-07-13 06:42:18,393 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-13 06:42:18,394 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/cluster_8d5ebd29-c8d7-9901-4c53-f6c110004b44, deleteOnExit=true 2023-07-13 06:42:18,394 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-13 06:42:18,394 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/test.cache.data in system properties and HBase conf 2023-07-13 06:42:18,395 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/hadoop.tmp.dir in system properties and HBase conf 2023-07-13 06:42:18,395 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/hadoop.log.dir in system properties and HBase conf 2023-07-13 06:42:18,396 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-13 06:42:18,396 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-13 06:42:18,397 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-13 06:42:18,493 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-13 06:42:18,818 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-13 06:42:18,821 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-13 06:42:18,822 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-13 06:42:18,822 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-13 06:42:18,822 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-13 06:42:18,822 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-13 06:42:18,823 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-13 06:42:18,823 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-13 06:42:18,823 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-13 06:42:18,824 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-13 06:42:18,824 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/nfs.dump.dir in system properties and HBase conf 2023-07-13 06:42:18,824 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/java.io.tmpdir in system properties and HBase conf 2023-07-13 06:42:18,824 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-13 06:42:18,825 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-13 06:42:18,825 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-13 06:42:19,251 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-13 06:42:19,254 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-13 06:42:19,494 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-13 06:42:19,644 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-13 06:42:19,658 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-13 06:42:19,694 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-13 06:42:19,724 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/java.io.tmpdir/Jetty_localhost_localdomain_43443_hdfs____.qxd1zm/webapp 2023-07-13 06:42:19,856 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:43443 2023-07-13 06:42:19,863 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-13 06:42:19,864 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-13 06:42:20,195 WARN [Listener at localhost.localdomain/46147] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-13 06:42:20,281 WARN [Listener at localhost.localdomain/46147] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-13 06:42:20,297 WARN [Listener at localhost.localdomain/46147] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-13 06:42:20,303 INFO [Listener at localhost.localdomain/46147] log.Slf4jLog(67): jetty-6.1.26 2023-07-13 06:42:20,307 INFO [Listener at localhost.localdomain/46147] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/java.io.tmpdir/Jetty_localhost_41659_datanode____.27ueak/webapp 2023-07-13 06:42:20,395 INFO [Listener at localhost.localdomain/46147] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41659 2023-07-13 06:42:20,678 WARN [Listener at localhost.localdomain/40339] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-13 06:42:20,688 WARN [Listener at localhost.localdomain/40339] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-13 06:42:20,691 WARN [Listener at localhost.localdomain/40339] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-13 06:42:20,693 INFO [Listener at localhost.localdomain/40339] log.Slf4jLog(67): jetty-6.1.26 2023-07-13 06:42:20,698 INFO [Listener at localhost.localdomain/40339] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/java.io.tmpdir/Jetty_localhost_34901_datanode____ygklsw/webapp 2023-07-13 06:42:20,791 INFO [Listener at localhost.localdomain/40339] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34901 2023-07-13 06:42:20,802 WARN [Listener at localhost.localdomain/40319] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-13 06:42:21,093 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x59d973904325c338: Processing first storage report for DS-9f07bcbe-b9b6-483b-b54a-1fd62e8b70d4 from datanode 305f01f8-519a-4fc6-a5d1-b59b788bbb0f 2023-07-13 06:42:21,095 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x59d973904325c338: from storage DS-9f07bcbe-b9b6-483b-b54a-1fd62e8b70d4 node DatanodeRegistration(127.0.0.1:46221, datanodeUuid=305f01f8-519a-4fc6-a5d1-b59b788bbb0f, infoPort=38751, infoSecurePort=0, ipcPort=40339, storageInfo=lv=-57;cid=testClusterID;nsid=1524037093;c=1689230539315), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-13 06:42:21,095 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x478481f2f1604010: Processing first storage report for DS-bd5046f7-7437-4b40-ab6d-2b891ce9bd15 from datanode cdebc699-f457-4676-abe1-0db4364b2ae5 2023-07-13 06:42:21,096 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x478481f2f1604010: from storage DS-bd5046f7-7437-4b40-ab6d-2b891ce9bd15 node DatanodeRegistration(127.0.0.1:32857, datanodeUuid=cdebc699-f457-4676-abe1-0db4364b2ae5, infoPort=34051, infoSecurePort=0, ipcPort=40319, storageInfo=lv=-57;cid=testClusterID;nsid=1524037093;c=1689230539315), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-13 06:42:21,096 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x59d973904325c338: Processing first storage report for DS-99676918-328f-44d8-8d40-a8b5a1b1fa98 from datanode 305f01f8-519a-4fc6-a5d1-b59b788bbb0f 2023-07-13 06:42:21,096 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x59d973904325c338: from storage DS-99676918-328f-44d8-8d40-a8b5a1b1fa98 node DatanodeRegistration(127.0.0.1:46221, datanodeUuid=305f01f8-519a-4fc6-a5d1-b59b788bbb0f, infoPort=38751, infoSecurePort=0, ipcPort=40339, storageInfo=lv=-57;cid=testClusterID;nsid=1524037093;c=1689230539315), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-13 06:42:21,096 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x478481f2f1604010: Processing first storage report for DS-ae009ffc-9cfe-4c98-be27-2cebda7310e7 from datanode cdebc699-f457-4676-abe1-0db4364b2ae5 2023-07-13 06:42:21,096 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x478481f2f1604010: from storage DS-ae009ffc-9cfe-4c98-be27-2cebda7310e7 node DatanodeRegistration(127.0.0.1:32857, datanodeUuid=cdebc699-f457-4676-abe1-0db4364b2ae5, infoPort=34051, infoSecurePort=0, ipcPort=40319, storageInfo=lv=-57;cid=testClusterID;nsid=1524037093;c=1689230539315), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-13 06:42:21,186 DEBUG [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370 2023-07-13 06:42:21,272 INFO [Listener at localhost.localdomain/40319] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/cluster_8d5ebd29-c8d7-9901-4c53-f6c110004b44/zookeeper_0, clientPort=58518, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/cluster_8d5ebd29-c8d7-9901-4c53-f6c110004b44/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/cluster_8d5ebd29-c8d7-9901-4c53-f6c110004b44/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-13 06:42:21,287 INFO [Listener at localhost.localdomain/40319] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58518 2023-07-13 06:42:21,299 INFO [Listener at localhost.localdomain/40319] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 06:42:21,302 INFO [Listener at localhost.localdomain/40319] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 06:42:21,941 INFO [Listener at localhost.localdomain/40319] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c with version=8 2023-07-13 06:42:21,941 INFO [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/hbase-staging 2023-07-13 06:42:22,216 INFO [Listener at localhost.localdomain/40319] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-13 06:42:22,602 INFO [Listener at localhost.localdomain/40319] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-07-13 06:42:22,638 INFO [Listener at localhost.localdomain/40319] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-13 06:42:22,639 INFO [Listener at localhost.localdomain/40319] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-13 06:42:22,639 INFO [Listener at localhost.localdomain/40319] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-13 06:42:22,639 INFO [Listener at localhost.localdomain/40319] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-13 06:42:22,639 INFO [Listener at localhost.localdomain/40319] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-13 06:42:22,767 INFO [Listener at localhost.localdomain/40319] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-13 06:42:22,824 DEBUG [Listener at localhost.localdomain/40319] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-13 06:42:22,902 INFO [Listener at localhost.localdomain/40319] ipc.NettyRpcServer(120): Bind to /148.251.75.209:44209 2023-07-13 06:42:22,910 INFO [Listener at localhost.localdomain/40319] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 06:42:22,912 INFO [Listener at localhost.localdomain/40319] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 06:42:22,934 INFO [Listener at localhost.localdomain/40319] zookeeper.RecoverableZooKeeper(93): Process identifier=master:44209 connecting to ZooKeeper ensemble=127.0.0.1:58518 2023-07-13 06:42:22,979 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:442090x0, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-13 06:42:22,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:44209-0x1008510fe3f0000 connected 2023-07-13 06:42:23,007 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ZKUtil(164): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-13 06:42:23,009 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ZKUtil(164): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-13 06:42:23,013 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ZKUtil(164): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-13 06:42:23,022 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44209 2023-07-13 06:42:23,023 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44209 2023-07-13 06:42:23,023 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44209 2023-07-13 06:42:23,024 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44209 2023-07-13 06:42:23,024 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44209 2023-07-13 06:42:23,029 INFO [Listener at localhost.localdomain/40319] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c, hbase.cluster.distributed=false 2023-07-13 06:42:23,089 INFO [Listener at localhost.localdomain/40319] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-13 06:42:23,089 INFO [Listener at localhost.localdomain/40319] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-13 06:42:23,089 INFO [Listener at localhost.localdomain/40319] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-13 06:42:23,089 INFO [Listener at localhost.localdomain/40319] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-13 06:42:23,089 INFO [Listener at localhost.localdomain/40319] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-13 06:42:23,089 INFO [Listener at localhost.localdomain/40319] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-13 06:42:23,093 INFO [Listener at localhost.localdomain/40319] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-13 06:42:23,096 INFO [Listener at localhost.localdomain/40319] ipc.NettyRpcServer(120): Bind to /148.251.75.209:39919 2023-07-13 06:42:23,098 INFO [Listener at localhost.localdomain/40319] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-13 06:42:23,104 DEBUG [Listener at localhost.localdomain/40319] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-13 06:42:23,105 INFO [Listener at localhost.localdomain/40319] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 06:42:23,107 INFO [Listener at localhost.localdomain/40319] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 06:42:23,108 INFO [Listener at localhost.localdomain/40319] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39919 connecting to ZooKeeper ensemble=127.0.0.1:58518 2023-07-13 06:42:23,112 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:399190x0, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-13 06:42:23,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:39919-0x1008510fe3f0001 connected 2023-07-13 06:42:23,113 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ZKUtil(164): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-13 06:42:23,114 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ZKUtil(164): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-13 06:42:23,115 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ZKUtil(164): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-13 06:42:23,116 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39919 2023-07-13 06:42:23,117 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39919 2023-07-13 06:42:23,117 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39919 2023-07-13 06:42:23,117 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39919 2023-07-13 06:42:23,118 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39919 2023-07-13 06:42:23,131 INFO [Listener at localhost.localdomain/40319] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-13 06:42:23,131 INFO [Listener at localhost.localdomain/40319] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-13 06:42:23,131 INFO [Listener at localhost.localdomain/40319] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-13 06:42:23,132 INFO [Listener at localhost.localdomain/40319] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-13 06:42:23,132 INFO [Listener at localhost.localdomain/40319] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-13 06:42:23,132 INFO [Listener at localhost.localdomain/40319] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-13 06:42:23,132 INFO [Listener at localhost.localdomain/40319] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-13 06:42:23,135 INFO [Listener at localhost.localdomain/40319] ipc.NettyRpcServer(120): Bind to /148.251.75.209:41937 2023-07-13 06:42:23,136 INFO [Listener at localhost.localdomain/40319] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-13 06:42:23,137 DEBUG [Listener at localhost.localdomain/40319] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-13 06:42:23,137 INFO [Listener at localhost.localdomain/40319] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 06:42:23,139 INFO [Listener at localhost.localdomain/40319] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 06:42:23,141 INFO [Listener at localhost.localdomain/40319] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41937 connecting to ZooKeeper ensemble=127.0.0.1:58518 2023-07-13 06:42:23,144 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:419370x0, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-13 06:42:23,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:41937-0x1008510fe3f0002 connected 2023-07-13 06:42:23,146 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ZKUtil(164): regionserver:41937-0x1008510fe3f0002, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-13 06:42:23,147 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ZKUtil(164): regionserver:41937-0x1008510fe3f0002, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-13 06:42:23,148 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ZKUtil(164): regionserver:41937-0x1008510fe3f0002, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-13 06:42:23,148 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41937 2023-07-13 06:42:23,148 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41937 2023-07-13 06:42:23,149 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41937 2023-07-13 06:42:23,149 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41937 2023-07-13 06:42:23,149 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41937 2023-07-13 06:42:23,150 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,44209,1689230542090 2023-07-13 06:42:23,160 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-13 06:42:23,162 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,44209,1689230542090 2023-07-13 06:42:23,181 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-13 06:42:23,181 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-13 06:42:23,181 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:41937-0x1008510fe3f0002, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-13 06:42:23,182 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 06:42:23,183 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-13 06:42:23,184 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,44209,1689230542090 from backup master directory 2023-07-13 06:42:23,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-13 06:42:23,186 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,44209,1689230542090 2023-07-13 06:42:23,187 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-13 06:42:23,187 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-13 06:42:23,187 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,44209,1689230542090 2023-07-13 06:42:23,190 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-13 06:42:23,191 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-13 06:42:23,272 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/hbase.id with ID: c61308e6-af66-4627-b6a4-80b47ec8901e 2023-07-13 06:42:23,327 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 06:42:23,344 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 06:42:23,387 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x757253c7 to 127.0.0.1:58518 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-13 06:42:23,417 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24771d05, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-13 06:42:23,438 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-13 06:42:23,440 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-13 06:42:23,456 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-13 06:42:23,456 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-13 06:42:23,458 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-13 06:42:23,462 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-13 06:42:23,463 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-13 06:42:23,492 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/MasterData/data/master/store-tmp 2023-07-13 06:42:23,522 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 06:42:23,522 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-13 06:42:23,522 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-13 06:42:23,522 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-13 06:42:23,523 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-13 06:42:23,523 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-13 06:42:23,523 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-13 06:42:23,523 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-13 06:42:23,525 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/MasterData/WALs/jenkins-hbase20.apache.org,44209,1689230542090 2023-07-13 06:42:23,546 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C44209%2C1689230542090, suffix=, logDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/MasterData/WALs/jenkins-hbase20.apache.org,44209,1689230542090, archiveDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/MasterData/oldWALs, maxLogs=10 2023-07-13 06:42:23,597 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32857,DS-bd5046f7-7437-4b40-ab6d-2b891ce9bd15,DISK] 2023-07-13 06:42:23,597 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46221,DS-9f07bcbe-b9b6-483b-b54a-1fd62e8b70d4,DISK] 2023-07-13 06:42:23,604 DEBUG [RS-EventLoopGroup-4-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-13 06:42:23,665 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/MasterData/WALs/jenkins-hbase20.apache.org,44209,1689230542090/jenkins-hbase20.apache.org%2C44209%2C1689230542090.1689230543556 2023-07-13 06:42:23,666 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32857,DS-bd5046f7-7437-4b40-ab6d-2b891ce9bd15,DISK], DatanodeInfoWithStorage[127.0.0.1:46221,DS-9f07bcbe-b9b6-483b-b54a-1fd62e8b70d4,DISK]] 2023-07-13 06:42:23,666 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-13 06:42:23,667 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 06:42:23,669 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-13 06:42:23,670 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-13 06:42:23,734 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-13 06:42:23,741 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-13 06:42:23,771 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-13 06:42:23,785 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 06:42:23,791 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-13 06:42:23,793 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-13 06:42:23,812 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-13 06:42:23,817 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-13 06:42:23,819 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61107089, jitterRate=-0.08943341672420502}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-13 06:42:23,819 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-13 06:42:23,820 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-13 06:42:23,842 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-13 06:42:23,842 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-13 06:42:23,844 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-13 06:42:23,846 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-13 06:42:23,872 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 26 msec 2023-07-13 06:42:23,872 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-13 06:42:23,893 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-13 06:42:23,898 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-13 06:42:23,919 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-13 06:42:23,922 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-13 06:42:23,924 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-13 06:42:23,928 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-13 06:42:23,932 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-13 06:42:23,934 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 06:42:23,936 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-13 06:42:23,936 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-13 06:42:23,946 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-13 06:42:23,953 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-13 06:42:23,954 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-13 06:42:23,954 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:41937-0x1008510fe3f0002, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-13 06:42:23,954 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 06:42:23,954 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,44209,1689230542090, sessionid=0x1008510fe3f0000, setting cluster-up flag (Was=false) 2023-07-13 06:42:23,969 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 06:42:23,973 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-13 06:42:23,975 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,44209,1689230542090 2023-07-13 06:42:23,978 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 06:42:23,982 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-13 06:42:23,983 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,44209,1689230542090 2023-07-13 06:42:23,985 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/.hbase-snapshot/.tmp 2023-07-13 06:42:24,054 INFO [RS:1;jenkins-hbase20:41937] regionserver.HRegionServer(951): ClusterId : c61308e6-af66-4627-b6a4-80b47ec8901e 2023-07-13 06:42:24,054 INFO [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(951): ClusterId : c61308e6-af66-4627-b6a4-80b47ec8901e 2023-07-13 06:42:24,061 DEBUG [RS:0;jenkins-hbase20:39919] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-13 06:42:24,061 DEBUG [RS:1;jenkins-hbase20:41937] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-13 06:42:24,067 DEBUG [RS:0;jenkins-hbase20:39919] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-13 06:42:24,067 DEBUG [RS:1;jenkins-hbase20:41937] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-13 06:42:24,067 DEBUG [RS:0;jenkins-hbase20:39919] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-13 06:42:24,067 DEBUG [RS:1;jenkins-hbase20:41937] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-13 06:42:24,070 DEBUG [RS:0;jenkins-hbase20:39919] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-13 06:42:24,070 DEBUG [RS:1;jenkins-hbase20:41937] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-13 06:42:24,073 DEBUG [RS:1;jenkins-hbase20:41937] zookeeper.ReadOnlyZKClient(139): Connect 0x7712866f to 127.0.0.1:58518 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-13 06:42:24,073 DEBUG [RS:0;jenkins-hbase20:39919] zookeeper.ReadOnlyZKClient(139): Connect 0x3954eb73 to 127.0.0.1:58518 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-13 06:42:24,086 DEBUG [RS:0;jenkins-hbase20:39919] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37931229, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-13 06:42:24,087 DEBUG [RS:0;jenkins-hbase20:39919] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33ac8003, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-13 06:42:24,089 DEBUG [RS:1;jenkins-hbase20:41937] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79d576fd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-13 06:42:24,089 DEBUG [RS:1;jenkins-hbase20:41937] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7deaed1f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-13 06:42:24,115 DEBUG [RS:1;jenkins-hbase20:41937] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase20:41937 2023-07-13 06:42:24,116 DEBUG [RS:0;jenkins-hbase20:39919] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:39919 2023-07-13 06:42:24,119 INFO [RS:1;jenkins-hbase20:41937] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-13 06:42:24,119 INFO [RS:0;jenkins-hbase20:39919] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-13 06:42:24,120 INFO [RS:1;jenkins-hbase20:41937] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-13 06:42:24,120 INFO [RS:0;jenkins-hbase20:39919] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-13 06:42:24,121 DEBUG [RS:1;jenkins-hbase20:41937] regionserver.HRegionServer(1022): About to register with Master. 2023-07-13 06:42:24,121 DEBUG [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(1022): About to register with Master. 2023-07-13 06:42:24,123 INFO [RS:1;jenkins-hbase20:41937] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,44209,1689230542090 with isa=jenkins-hbase20.apache.org/148.251.75.209:41937, startcode=1689230543130 2023-07-13 06:42:24,123 INFO [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,44209,1689230542090 with isa=jenkins-hbase20.apache.org/148.251.75.209:39919, startcode=1689230543088 2023-07-13 06:42:24,135 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-13 06:42:24,142 DEBUG [RS:1;jenkins-hbase20:41937] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-13 06:42:24,142 DEBUG [RS:0;jenkins-hbase20:39919] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-13 06:42:24,147 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-13 06:42:24,148 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-13 06:42:24,148 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-13 06:42:24,148 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-13 06:42:24,148 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-07-13 06:42:24,148 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,149 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-13 06:42:24,149 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,154 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689230574153 2023-07-13 06:42:24,156 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-13 06:42:24,158 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-13 06:42:24,158 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-13 06:42:24,168 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-13 06:42:24,176 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-13 06:42:24,184 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-13 06:42:24,185 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-13 06:42:24,185 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-13 06:42:24,185 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-13 06:42:24,186 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:24,188 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-13 06:42:24,190 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-13 06:42:24,191 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-13 06:42:24,194 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-13 06:42:24,194 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-13 06:42:24,197 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689230544196,5,FailOnTimeoutGroup] 2023-07-13 06:42:24,197 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689230544197,5,FailOnTimeoutGroup] 2023-07-13 06:42:24,198 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:24,198 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-13 06:42:24,202 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:24,202 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:24,207 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34379, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-13 06:42:24,207 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:51427, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-13 06:42:24,208 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-13 06:42:24,209 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-13 06:42:24,210 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c 2023-07-13 06:42:24,226 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44209] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:24,232 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 06:42:24,235 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-13 06:42:24,238 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/info 2023-07-13 06:42:24,238 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-13 06:42:24,240 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 06:42:24,240 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-13 06:42:24,241 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44209] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,41937,1689230543130 2023-07-13 06:42:24,244 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/rep_barrier 2023-07-13 06:42:24,244 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-13 06:42:24,245 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 06:42:24,245 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-13 06:42:24,246 DEBUG [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c 2023-07-13 06:42:24,246 DEBUG [RS:1;jenkins-hbase20:41937] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c 2023-07-13 06:42:24,247 DEBUG [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:46147 2023-07-13 06:42:24,247 DEBUG [RS:1;jenkins-hbase20:41937] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:46147 2023-07-13 06:42:24,247 DEBUG [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-13 06:42:24,247 DEBUG [RS:1;jenkins-hbase20:41937] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-13 06:42:24,248 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/table 2023-07-13 06:42:24,249 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-13 06:42:24,250 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 06:42:24,252 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740 2023-07-13 06:42:24,252 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740 2023-07-13 06:42:24,253 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 06:42:24,254 DEBUG [RS:0;jenkins-hbase20:39919] zookeeper.ZKUtil(162): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:24,254 DEBUG [RS:1;jenkins-hbase20:41937] zookeeper.ZKUtil(162): regionserver:41937-0x1008510fe3f0002, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,41937,1689230543130 2023-07-13 06:42:24,254 WARN [RS:0;jenkins-hbase20:39919] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-13 06:42:24,254 WARN [RS:1;jenkins-hbase20:41937] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-13 06:42:24,255 INFO [RS:0;jenkins-hbase20:39919] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-13 06:42:24,256 INFO [RS:1;jenkins-hbase20:41937] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-13 06:42:24,256 DEBUG [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:24,256 DEBUG [RS:1;jenkins-hbase20:41937] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130 2023-07-13 06:42:24,257 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,41937,1689230543130] 2023-07-13 06:42:24,257 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,39919,1689230543088] 2023-07-13 06:42:24,260 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-13 06:42:24,264 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-13 06:42:24,271 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-13 06:42:24,272 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70537240, jitterRate=0.05108678340911865}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-13 06:42:24,272 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-13 06:42:24,272 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-13 06:42:24,272 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-13 06:42:24,272 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-13 06:42:24,272 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-13 06:42:24,272 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-13 06:42:24,273 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-13 06:42:24,274 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-13 06:42:24,274 DEBUG [RS:0;jenkins-hbase20:39919] zookeeper.ZKUtil(162): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:24,274 DEBUG [RS:1;jenkins-hbase20:41937] zookeeper.ZKUtil(162): regionserver:41937-0x1008510fe3f0002, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:24,275 DEBUG [RS:0;jenkins-hbase20:39919] zookeeper.ZKUtil(162): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,41937,1689230543130 2023-07-13 06:42:24,275 DEBUG [RS:1;jenkins-hbase20:41937] zookeeper.ZKUtil(162): regionserver:41937-0x1008510fe3f0002, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,41937,1689230543130 2023-07-13 06:42:24,279 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-13 06:42:24,279 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-13 06:42:24,285 DEBUG [RS:0;jenkins-hbase20:39919] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-13 06:42:24,285 DEBUG [RS:1;jenkins-hbase20:41937] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-13 06:42:24,287 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-13 06:42:24,296 INFO [RS:1;jenkins-hbase20:41937] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-13 06:42:24,296 INFO [RS:0;jenkins-hbase20:39919] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-13 06:42:24,301 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-13 06:42:24,306 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-13 06:42:24,317 INFO [RS:1;jenkins-hbase20:41937] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-13 06:42:24,318 INFO [RS:0;jenkins-hbase20:39919] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-13 06:42:24,321 INFO [RS:0;jenkins-hbase20:39919] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-13 06:42:24,321 INFO [RS:1;jenkins-hbase20:41937] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-13 06:42:24,321 INFO [RS:0;jenkins-hbase20:39919] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:24,322 INFO [RS:1;jenkins-hbase20:41937] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:24,323 INFO [RS:1;jenkins-hbase20:41937] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-13 06:42:24,323 INFO [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-13 06:42:24,330 INFO [RS:1;jenkins-hbase20:41937] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:24,330 INFO [RS:0;jenkins-hbase20:39919] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:24,330 DEBUG [RS:1;jenkins-hbase20:41937] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,330 DEBUG [RS:0;jenkins-hbase20:39919] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,331 DEBUG [RS:1;jenkins-hbase20:41937] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,331 DEBUG [RS:0;jenkins-hbase20:39919] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,331 DEBUG [RS:1;jenkins-hbase20:41937] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,331 DEBUG [RS:0;jenkins-hbase20:39919] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,331 DEBUG [RS:1;jenkins-hbase20:41937] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,331 DEBUG [RS:0;jenkins-hbase20:39919] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,331 DEBUG [RS:1;jenkins-hbase20:41937] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,331 DEBUG [RS:0;jenkins-hbase20:39919] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,331 DEBUG [RS:1;jenkins-hbase20:41937] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-13 06:42:24,331 DEBUG [RS:0;jenkins-hbase20:39919] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-13 06:42:24,331 DEBUG [RS:1;jenkins-hbase20:41937] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,331 DEBUG [RS:0;jenkins-hbase20:39919] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,331 DEBUG [RS:1;jenkins-hbase20:41937] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,331 DEBUG [RS:0;jenkins-hbase20:39919] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,331 DEBUG [RS:1;jenkins-hbase20:41937] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,331 DEBUG [RS:0;jenkins-hbase20:39919] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,331 DEBUG [RS:1;jenkins-hbase20:41937] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,332 DEBUG [RS:0;jenkins-hbase20:39919] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:24,334 INFO [RS:1;jenkins-hbase20:41937] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:24,334 INFO [RS:1;jenkins-hbase20:41937] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:24,334 INFO [RS:0;jenkins-hbase20:39919] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:24,335 INFO [RS:1;jenkins-hbase20:41937] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:24,335 INFO [RS:0;jenkins-hbase20:39919] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:24,335 INFO [RS:0;jenkins-hbase20:39919] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:24,349 INFO [RS:1;jenkins-hbase20:41937] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-13 06:42:24,349 INFO [RS:0;jenkins-hbase20:39919] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-13 06:42:24,351 INFO [RS:0;jenkins-hbase20:39919] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,39919,1689230543088-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:24,351 INFO [RS:1;jenkins-hbase20:41937] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,41937,1689230543130-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:24,366 INFO [RS:0;jenkins-hbase20:39919] regionserver.Replication(203): jenkins-hbase20.apache.org,39919,1689230543088 started 2023-07-13 06:42:24,366 INFO [RS:1;jenkins-hbase20:41937] regionserver.Replication(203): jenkins-hbase20.apache.org,41937,1689230543130 started 2023-07-13 06:42:24,366 INFO [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,39919,1689230543088, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:39919, sessionid=0x1008510fe3f0001 2023-07-13 06:42:24,366 INFO [RS:1;jenkins-hbase20:41937] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,41937,1689230543130, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:41937, sessionid=0x1008510fe3f0002 2023-07-13 06:42:24,366 DEBUG [RS:0;jenkins-hbase20:39919] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-13 06:42:24,366 DEBUG [RS:1;jenkins-hbase20:41937] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-13 06:42:24,366 DEBUG [RS:0;jenkins-hbase20:39919] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:24,367 DEBUG [RS:1;jenkins-hbase20:41937] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,41937,1689230543130 2023-07-13 06:42:24,367 DEBUG [RS:0;jenkins-hbase20:39919] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,39919,1689230543088' 2023-07-13 06:42:24,367 DEBUG [RS:1;jenkins-hbase20:41937] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,41937,1689230543130' 2023-07-13 06:42:24,368 DEBUG [RS:1;jenkins-hbase20:41937] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-13 06:42:24,368 DEBUG [RS:0;jenkins-hbase20:39919] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-13 06:42:24,369 DEBUG [RS:0;jenkins-hbase20:39919] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-13 06:42:24,369 DEBUG [RS:1;jenkins-hbase20:41937] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-13 06:42:24,369 DEBUG [RS:0;jenkins-hbase20:39919] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-13 06:42:24,369 DEBUG [RS:0;jenkins-hbase20:39919] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-13 06:42:24,369 DEBUG [RS:0;jenkins-hbase20:39919] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:24,369 DEBUG [RS:0;jenkins-hbase20:39919] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,39919,1689230543088' 2023-07-13 06:42:24,369 DEBUG [RS:0;jenkins-hbase20:39919] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-13 06:42:24,369 DEBUG [RS:1;jenkins-hbase20:41937] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-13 06:42:24,369 DEBUG [RS:1;jenkins-hbase20:41937] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-13 06:42:24,369 DEBUG [RS:1;jenkins-hbase20:41937] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,41937,1689230543130 2023-07-13 06:42:24,370 DEBUG [RS:1;jenkins-hbase20:41937] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,41937,1689230543130' 2023-07-13 06:42:24,370 DEBUG [RS:1;jenkins-hbase20:41937] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-13 06:42:24,370 DEBUG [RS:0;jenkins-hbase20:39919] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-13 06:42:24,370 DEBUG [RS:1;jenkins-hbase20:41937] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-13 06:42:24,371 DEBUG [RS:0;jenkins-hbase20:39919] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-13 06:42:24,371 INFO [RS:0;jenkins-hbase20:39919] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-13 06:42:24,371 INFO [RS:0;jenkins-hbase20:39919] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-13 06:42:24,371 DEBUG [RS:1;jenkins-hbase20:41937] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-13 06:42:24,371 INFO [RS:1;jenkins-hbase20:41937] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-13 06:42:24,371 INFO [RS:1;jenkins-hbase20:41937] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-13 06:42:24,459 DEBUG [jenkins-hbase20:44209] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-13 06:42:24,464 DEBUG [jenkins-hbase20:44209] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-13 06:42:24,472 DEBUG [jenkins-hbase20:44209] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-13 06:42:24,472 DEBUG [jenkins-hbase20:44209] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-13 06:42:24,472 DEBUG [jenkins-hbase20:44209] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-13 06:42:24,475 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,39919,1689230543088, state=OPENING 2023-07-13 06:42:24,481 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-13 06:42:24,482 INFO [RS:1;jenkins-hbase20:41937] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C41937%2C1689230543130, suffix=, logDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130, archiveDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/oldWALs, maxLogs=32 2023-07-13 06:42:24,482 INFO [RS:0;jenkins-hbase20:39919] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C39919%2C1689230543088, suffix=, logDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088, archiveDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/oldWALs, maxLogs=32 2023-07-13 06:42:24,483 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 06:42:24,484 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-13 06:42:24,488 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,39919,1689230543088}] 2023-07-13 06:42:24,514 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46221,DS-9f07bcbe-b9b6-483b-b54a-1fd62e8b70d4,DISK] 2023-07-13 06:42:24,514 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46221,DS-9f07bcbe-b9b6-483b-b54a-1fd62e8b70d4,DISK] 2023-07-13 06:42:24,514 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32857,DS-bd5046f7-7437-4b40-ab6d-2b891ce9bd15,DISK] 2023-07-13 06:42:24,518 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32857,DS-bd5046f7-7437-4b40-ab6d-2b891ce9bd15,DISK] 2023-07-13 06:42:24,525 INFO [RS:0;jenkins-hbase20:39919] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088/jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485 2023-07-13 06:42:24,526 INFO [RS:1;jenkins-hbase20:41937] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130/jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485 2023-07-13 06:42:24,526 DEBUG [RS:0;jenkins-hbase20:39919] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32857,DS-bd5046f7-7437-4b40-ab6d-2b891ce9bd15,DISK], DatanodeInfoWithStorage[127.0.0.1:46221,DS-9f07bcbe-b9b6-483b-b54a-1fd62e8b70d4,DISK]] 2023-07-13 06:42:24,526 DEBUG [RS:1;jenkins-hbase20:41937] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46221,DS-9f07bcbe-b9b6-483b-b54a-1fd62e8b70d4,DISK], DatanodeInfoWithStorage[127.0.0.1:32857,DS-bd5046f7-7437-4b40-ab6d-2b891ce9bd15,DISK]] 2023-07-13 06:42:24,696 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:24,697 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-13 06:42:24,701 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47912, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-13 06:42:24,715 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-13 06:42:24,715 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-13 06:42:24,719 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088, archiveDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/oldWALs, maxLogs=32 2023-07-13 06:42:24,735 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46221,DS-9f07bcbe-b9b6-483b-b54a-1fd62e8b70d4,DISK] 2023-07-13 06:42:24,736 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32857,DS-bd5046f7-7437-4b40-ab6d-2b891ce9bd15,DISK] 2023-07-13 06:42:24,744 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088/jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta 2023-07-13 06:42:24,745 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46221,DS-9f07bcbe-b9b6-483b-b54a-1fd62e8b70d4,DISK], DatanodeInfoWithStorage[127.0.0.1:32857,DS-bd5046f7-7437-4b40-ab6d-2b891ce9bd15,DISK]] 2023-07-13 06:42:24,745 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-13 06:42:24,747 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-13 06:42:24,765 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-13 06:42:24,798 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-13 06:42:24,802 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-13 06:42:24,802 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 06:42:24,802 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-13 06:42:24,802 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-13 06:42:24,805 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-13 06:42:24,807 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/info 2023-07-13 06:42:24,807 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/info 2023-07-13 06:42:24,808 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-13 06:42:24,809 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 06:42:24,809 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-13 06:42:24,810 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/rep_barrier 2023-07-13 06:42:24,810 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/rep_barrier 2023-07-13 06:42:24,811 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-13 06:42:24,811 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 06:42:24,812 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-13 06:42:24,813 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/table 2023-07-13 06:42:24,813 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/table 2023-07-13 06:42:24,813 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-13 06:42:24,814 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 06:42:24,816 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740 2023-07-13 06:42:24,818 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740 2023-07-13 06:42:24,821 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-13 06:42:24,824 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-13 06:42:24,825 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64409714, jitterRate=-0.0402204692363739}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-13 06:42:24,825 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-13 06:42:24,834 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689230544690 2023-07-13 06:42:24,854 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-13 06:42:24,855 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-13 06:42:24,856 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,39919,1689230543088, state=OPEN 2023-07-13 06:42:24,859 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-13 06:42:24,860 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-13 06:42:24,868 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-13 06:42:24,869 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,39919,1689230543088 in 372 msec 2023-07-13 06:42:24,877 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-13 06:42:24,877 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 583 msec 2023-07-13 06:42:24,887 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 836 msec 2023-07-13 06:42:24,888 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689230544887, completionTime=-1 2023-07-13 06:42:24,888 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-13 06:42:24,889 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-13 06:42:24,958 DEBUG [hconnection-0x227d375d-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 06:42:24,961 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47920, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 06:42:24,978 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-13 06:42:24,978 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689230604978 2023-07-13 06:42:24,978 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689230664978 2023-07-13 06:42:24,978 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 89 msec 2023-07-13 06:42:25,003 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44209,1689230542090-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:25,004 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44209,1689230542090-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:25,004 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44209,1689230542090-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:25,005 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:44209, period=300000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:25,005 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:25,011 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-13 06:42:25,018 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-13 06:42:25,020 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-13 06:42:25,028 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-13 06:42:25,031 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-13 06:42:25,034 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-13 06:42:25,053 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/.tmp/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:25,055 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/.tmp/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce empty. 2023-07-13 06:42:25,056 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/.tmp/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:25,057 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-13 06:42:25,104 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-13 06:42:25,107 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => a76e4d3d13ae5bb10f9d5a76e91300ce, NAME => 'hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/.tmp 2023-07-13 06:42:25,125 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 06:42:25,126 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing a76e4d3d13ae5bb10f9d5a76e91300ce, disabling compactions & flushes 2023-07-13 06:42:25,126 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:25,126 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:25,126 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. after waiting 0 ms 2023-07-13 06:42:25,126 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:25,126 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:25,126 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for a76e4d3d13ae5bb10f9d5a76e91300ce: 2023-07-13 06:42:25,131 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-13 06:42:25,146 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689230545134"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689230545134"}]},"ts":"1689230545134"} 2023-07-13 06:42:25,179 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-13 06:42:25,182 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-13 06:42:25,186 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689230545182"}]},"ts":"1689230545182"} 2023-07-13 06:42:25,191 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-13 06:42:25,194 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-13 06:42:25,197 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-13 06:42:25,197 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-13 06:42:25,197 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-13 06:42:25,200 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a76e4d3d13ae5bb10f9d5a76e91300ce, ASSIGN}] 2023-07-13 06:42:25,204 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a76e4d3d13ae5bb10f9d5a76e91300ce, ASSIGN 2023-07-13 06:42:25,207 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=a76e4d3d13ae5bb10f9d5a76e91300ce, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39919,1689230543088; forceNewPlan=false, retain=false 2023-07-13 06:42:25,363 INFO [jenkins-hbase20:44209] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-13 06:42:25,366 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=a76e4d3d13ae5bb10f9d5a76e91300ce, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:25,367 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689230545365"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689230545365"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689230545365"}]},"ts":"1689230545365"} 2023-07-13 06:42:25,373 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure a76e4d3d13ae5bb10f9d5a76e91300ce, server=jenkins-hbase20.apache.org,39919,1689230543088}] 2023-07-13 06:42:25,536 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:25,536 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a76e4d3d13ae5bb10f9d5a76e91300ce, NAME => 'hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce.', STARTKEY => '', ENDKEY => ''} 2023-07-13 06:42:25,538 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:25,538 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 06:42:25,538 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:25,538 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:25,541 INFO [StoreOpener-a76e4d3d13ae5bb10f9d5a76e91300ce-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:25,544 DEBUG [StoreOpener-a76e4d3d13ae5bb10f9d5a76e91300ce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/info 2023-07-13 06:42:25,544 DEBUG [StoreOpener-a76e4d3d13ae5bb10f9d5a76e91300ce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/info 2023-07-13 06:42:25,545 INFO [StoreOpener-a76e4d3d13ae5bb10f9d5a76e91300ce-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a76e4d3d13ae5bb10f9d5a76e91300ce columnFamilyName info 2023-07-13 06:42:25,546 INFO [StoreOpener-a76e4d3d13ae5bb10f9d5a76e91300ce-1] regionserver.HStore(310): Store=a76e4d3d13ae5bb10f9d5a76e91300ce/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 06:42:25,548 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:25,549 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:25,554 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:25,557 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-13 06:42:25,558 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened a76e4d3d13ae5bb10f9d5a76e91300ce; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59660621, jitterRate=-0.11098746955394745}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-13 06:42:25,558 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for a76e4d3d13ae5bb10f9d5a76e91300ce: 2023-07-13 06:42:25,560 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce., pid=6, masterSystemTime=1689230545527 2023-07-13 06:42:25,564 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:25,564 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:25,566 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=a76e4d3d13ae5bb10f9d5a76e91300ce, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:25,566 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689230545565"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689230545565"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689230545565"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689230545565"}]},"ts":"1689230545565"} 2023-07-13 06:42:25,576 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-13 06:42:25,576 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure a76e4d3d13ae5bb10f9d5a76e91300ce, server=jenkins-hbase20.apache.org,39919,1689230543088 in 199 msec 2023-07-13 06:42:25,579 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-13 06:42:25,580 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=a76e4d3d13ae5bb10f9d5a76e91300ce, ASSIGN in 376 msec 2023-07-13 06:42:25,581 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-13 06:42:25,582 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689230545581"}]},"ts":"1689230545581"} 2023-07-13 06:42:25,586 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-13 06:42:25,589 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-13 06:42:25,592 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 568 msec 2023-07-13 06:42:25,632 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-13 06:42:25,633 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-13 06:42:25,634 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 06:42:25,674 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-13 06:42:25,693 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-13 06:42:25,699 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 31 msec 2023-07-13 06:42:25,708 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-13 06:42:25,718 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-13 06:42:25,722 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 14 msec 2023-07-13 06:42:25,738 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-13 06:42:25,740 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-13 06:42:25,741 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.553sec 2023-07-13 06:42:25,745 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-13 06:42:25,748 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-13 06:42:25,748 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-13 06:42:25,749 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44209,1689230542090-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-13 06:42:25,750 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44209,1689230542090-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-13 06:42:25,761 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ReadOnlyZKClient(139): Connect 0x78e7a6a9 to 127.0.0.1:58518 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-13 06:42:25,761 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-13 06:42:25,769 DEBUG [Listener at localhost.localdomain/40319] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@208e4664, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-13 06:42:25,790 DEBUG [hconnection-0x2a075ac-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 06:42:25,801 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47922, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 06:42:25,810 INFO [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,44209,1689230542090 2023-07-13 06:42:25,819 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-13 06:42:25,822 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48212, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-13 06:42:25,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44209] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-13 06:42:25,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44209] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-13 06:42:25,838 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-13 06:42:25,841 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-13 06:42:25,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44209] master.MasterRpcServices(700): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-13 06:42:25,843 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/.tmp/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:25,844 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/.tmp/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba empty. 2023-07-13 06:42:25,847 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/.tmp/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:25,847 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-13 06:42:25,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44209] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-13 06:42:25,872 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-13 06:42:25,874 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => fc3ebf4b2149ca5230c85524c0de4bba, NAME => 't,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/.tmp 2023-07-13 06:42:25,892 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 06:42:25,893 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing fc3ebf4b2149ca5230c85524c0de4bba, disabling compactions & flushes 2023-07-13 06:42:25,893 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:25,893 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:25,893 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. after waiting 0 ms 2023-07-13 06:42:25,893 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:25,893 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:25,893 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for fc3ebf4b2149ca5230c85524c0de4bba: 2023-07-13 06:42:25,897 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-13 06:42:25,899 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689230545899"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689230545899"}]},"ts":"1689230545899"} 2023-07-13 06:42:25,902 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-13 06:42:25,903 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-13 06:42:25,903 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689230545903"}]},"ts":"1689230545903"} 2023-07-13 06:42:25,905 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-13 06:42:25,908 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-13 06:42:25,909 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-13 06:42:25,909 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-13 06:42:25,909 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-13 06:42:25,910 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=fc3ebf4b2149ca5230c85524c0de4bba, ASSIGN}] 2023-07-13 06:42:25,912 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=fc3ebf4b2149ca5230c85524c0de4bba, ASSIGN 2023-07-13 06:42:25,913 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=fc3ebf4b2149ca5230c85524c0de4bba, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41937,1689230543130; forceNewPlan=false, retain=false 2023-07-13 06:42:25,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44209] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-13 06:42:26,063 INFO [jenkins-hbase20:44209] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-13 06:42:26,065 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=fc3ebf4b2149ca5230c85524c0de4bba, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41937,1689230543130 2023-07-13 06:42:26,066 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689230546065"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689230546065"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689230546065"}]},"ts":"1689230546065"} 2023-07-13 06:42:26,074 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure fc3ebf4b2149ca5230c85524c0de4bba, server=jenkins-hbase20.apache.org,41937,1689230543130}] 2023-07-13 06:42:26,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44209] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-13 06:42:26,229 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,41937,1689230543130 2023-07-13 06:42:26,229 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-13 06:42:26,232 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:56724, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-13 06:42:26,239 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:26,239 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fc3ebf4b2149ca5230c85524c0de4bba, NAME => 't,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.', STARTKEY => '', ENDKEY => ''} 2023-07-13 06:42:26,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:26,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 06:42:26,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:26,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:26,255 INFO [StoreOpener-fc3ebf4b2149ca5230c85524c0de4bba-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:26,258 DEBUG [StoreOpener-fc3ebf4b2149ca5230c85524c0de4bba-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/f 2023-07-13 06:42:26,258 DEBUG [StoreOpener-fc3ebf4b2149ca5230c85524c0de4bba-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/f 2023-07-13 06:42:26,259 INFO [StoreOpener-fc3ebf4b2149ca5230c85524c0de4bba-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fc3ebf4b2149ca5230c85524c0de4bba columnFamilyName f 2023-07-13 06:42:26,260 INFO [StoreOpener-fc3ebf4b2149ca5230c85524c0de4bba-1] regionserver.HStore(310): Store=fc3ebf4b2149ca5230c85524c0de4bba/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 06:42:26,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:26,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:26,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:26,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-13 06:42:26,278 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened fc3ebf4b2149ca5230c85524c0de4bba; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71423189, jitterRate=0.06428845226764679}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-13 06:42:26,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for fc3ebf4b2149ca5230c85524c0de4bba: 2023-07-13 06:42:26,280 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba., pid=11, masterSystemTime=1689230546228 2023-07-13 06:42:26,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:26,286 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:26,287 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=fc3ebf4b2149ca5230c85524c0de4bba, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41937,1689230543130 2023-07-13 06:42:26,287 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689230546287"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689230546287"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689230546287"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689230546287"}]},"ts":"1689230546287"} 2023-07-13 06:42:26,294 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-13 06:42:26,294 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure fc3ebf4b2149ca5230c85524c0de4bba, server=jenkins-hbase20.apache.org,41937,1689230543130 in 218 msec 2023-07-13 06:42:26,298 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-13 06:42:26,298 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=fc3ebf4b2149ca5230c85524c0de4bba, ASSIGN in 384 msec 2023-07-13 06:42:26,300 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-13 06:42:26,300 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689230546300"}]},"ts":"1689230546300"} 2023-07-13 06:42:26,303 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-13 06:42:26,312 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-13 06:42:26,315 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 479 msec 2023-07-13 06:42:26,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44209] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-13 06:42:26,469 INFO [Listener at localhost.localdomain/40319] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-13 06:42:26,469 DEBUG [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-13 06:42:26,472 INFO [Listener at localhost.localdomain/40319] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-13 06:42:26,479 INFO [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-13 06:42:26,480 INFO [Listener at localhost.localdomain/40319] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-13 06:42:26,480 INFO [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-13 06:42:26,484 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 06:42:26,487 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:56738, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 06:42:26,547 INFO [Listener at localhost.localdomain/40319] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=296, OpenFileDescriptor=585, MaxFileDescriptor=60000, SystemLoadAverage=249, ProcessCount=168, AvailableMemoryMB=6281 2023-07-13 06:42:26,549 INFO [Listener at localhost.localdomain/40319] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-13 06:42:26,555 DEBUG [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-13 06:42:26,556 DEBUG [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:26,556 INFO [Listener at localhost.localdomain/40319] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-13 06:42:26,556 INFO [Listener at localhost.localdomain/40319] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-13 06:42:26,566 INFO [Listener at localhost.localdomain/40319] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689230545581/Put/vlen=2/seqid=0} 2023-07-13 06:42:26,567 INFO [Listener at localhost.localdomain/40319] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-13 06:42:26,567 INFO [Listener at localhost.localdomain/40319] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce./info:regioninfo/1689230545565/Put/vlen=41/seqid=0, hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce./info:seqnumDuringOpen/1689230545565/Put/vlen=8/seqid=0, hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce./info:server/1689230545565/Put/vlen=32/seqid=0, hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce./info:serverstartcode/1689230545565/Put/vlen=8/seqid=0, hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce./info:sn/1689230545365/Put/vlen=46/seqid=0, hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce./info:state/1689230545565/Put/vlen=4/seqid=0} 2023-07-13 06:42:26,568 INFO [Listener at localhost.localdomain/40319] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => a76e4d3d13ae5bb10f9d5a76e91300ce, NAME => 'hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce.', STARTKEY => '', ENDKEY => ''} 2023-07-13 06:42:26,568 INFO [Listener at localhost.localdomain/40319] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689230546300/Put/vlen=2/seqid=0} 2023-07-13 06:42:26,568 INFO [Listener at localhost.localdomain/40319] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-13 06:42:26,568 INFO [Listener at localhost.localdomain/40319] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba./info:regioninfo/1689230546287/Put/vlen=35/seqid=0, t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba./info:seqnumDuringOpen/1689230546287/Put/vlen=8/seqid=0, t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba./info:server/1689230546287/Put/vlen=32/seqid=0, t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba./info:serverstartcode/1689230546287/Put/vlen=8/seqid=0, t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba./info:sn/1689230546065/Put/vlen=46/seqid=0, t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba./info:state/1689230546287/Put/vlen=4/seqid=0} 2023-07-13 06:42:26,568 INFO [Listener at localhost.localdomain/40319] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => fc3ebf4b2149ca5230c85524c0de4bba, NAME => 't,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.', STARTKEY => '', ENDKEY => ''} 2023-07-13 06:42:26,570 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ReadOnlyZKClient(139): Connect 0x39c41235 to 127.0.0.1:58518 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-13 06:42:26,574 DEBUG [Listener at localhost.localdomain/40319] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c805c6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-13 06:42:26,575 INFO [Listener at localhost.localdomain/40319] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-13 06:42:26,575 INFO [Listener at localhost.localdomain/40319] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-13 06:42:26,579 DEBUG [hconnection-0x69aa34d9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 06:42:26,592 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47934, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 06:42:26,597 DEBUG [hconnection-0x69aa34d9-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 06:42:26,601 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:56742, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 06:42:26,603 INFO [Listener at localhost.localdomain/40319] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-13 06:42:26,606 ERROR [Listener at localhost.localdomain/40319] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,41937,1689230543130: die! ***** 2023-07-13 06:42:26,607 ERROR [Listener at localhost.localdomain/40319] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-13 06:42:26,610 DEBUG [Listener at localhost.localdomain/40319] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-13 06:42:26,614 DEBUG [Listener at localhost.localdomain/40319] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-13 06:42:26,614 DEBUG [Listener at localhost.localdomain/40319] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-13 06:42:26,615 DEBUG [Listener at localhost.localdomain/40319] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-13 06:42:26,616 INFO [Listener at localhost.localdomain/40319] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 902299648, "init": 524288000, "max": 2051014656, "used": 393518384 }, "NonHeapMemoryUsage": { "committed": 113074176, "init": 2555904, "max": -1, "used": 111111656 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org" } ] } 2023-07-13 06:42:26,621 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=44209] master.MasterRpcServices(612): jenkins-hbase20.apache.org,41937,1689230543130 reported a fatal error: ***** ABORTING region server jenkins-hbase20.apache.org,41937,1689230543130: die! ***** 2023-07-13 06:42:26,624 INFO [Listener at localhost.localdomain/40319] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,41937,1689230543130' ***** 2023-07-13 06:42:26,625 INFO [Listener at localhost.localdomain/40319] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-13 06:42:26,625 INFO [RS:1;jenkins-hbase20:41937] regionserver.HeapMemoryManager(220): Stopping 2023-07-13 06:42:26,625 INFO [RS:1;jenkins-hbase20:41937] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-13 06:42:26,625 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-13 06:42:26,625 INFO [RS:1;jenkins-hbase20:41937] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-13 06:42:26,626 INFO [RS:1;jenkins-hbase20:41937] regionserver.HRegionServer(3305): Received CLOSE for fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:26,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41937] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner -5640695203988766719 2023-07-13 06:42:26,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41937] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,41937,1689230543130 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-13 06:42:26,627 INFO [RS:1;jenkins-hbase20:41937] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,41937,1689230543130 2023-07-13 06:42:26,628 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing fc3ebf4b2149ca5230c85524c0de4bba, disabling compactions & flushes 2023-07-13 06:42:26,628 DEBUG [RS:1;jenkins-hbase20:41937] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7712866f to 127.0.0.1:58518 2023-07-13 06:42:26,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41937] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 148.251.75.209:56742 deadline: 1689230606626, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,41937,1689230543130 aborting 2023-07-13 06:42:26,628 DEBUG [RS:1;jenkins-hbase20:41937] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 06:42:26,628 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:26,628 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:26,628 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. after waiting 0 ms 2023-07-13 06:42:26,628 INFO [RS:1;jenkins-hbase20:41937] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-13 06:42:26,628 DEBUG [RS:1;jenkins-hbase20:41937] regionserver.HRegionServer(1478): Online Regions={fc3ebf4b2149ca5230c85524c0de4bba=t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.} 2023-07-13 06:42:26,628 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:26,629 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:26,630 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:26,630 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for fc3ebf4b2149ca5230c85524c0de4bba: 2023-07-13 06:42:26,630 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:26,631 DEBUG [RS:1;jenkins-hbase20:41937] regionserver.HRegionServer(1504): Waiting on 2023-07-13 06:42:26,645 DEBUG [Listener at localhost.localdomain/40319] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,41937,1689230543130 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,41937,1689230543130 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-13 06:42:26,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41937] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,41937,1689230543130 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-13 06:42:26,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41937] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 148.251.75.209:56742 deadline: 1689230606647, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,41937,1689230543130 aborting 2023-07-13 06:42:26,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41937] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,41937,1689230543130 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-13 06:42:26,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41937] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Scan size: 125 connection: 148.251.75.209:56742 deadline: 1689230606763, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,41937,1689230543130 aborting 2023-07-13 06:42:26,831 INFO [RS:1;jenkins-hbase20:41937] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,41937,1689230543130; all regions closed. 2023-07-13 06:42:26,848 DEBUG [RS:1;jenkins-hbase20:41937] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 06:42:26,849 INFO [RS:1;jenkins-hbase20:41937] regionserver.LeaseManager(133): Closed leases 2023-07-13 06:42:26,849 INFO [RS:1;jenkins-hbase20:41937] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-13 06:42:26,849 INFO [RS:1;jenkins-hbase20:41937] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-13 06:42:26,849 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-13 06:42:26,849 INFO [RS:1;jenkins-hbase20:41937] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-13 06:42:26,849 INFO [RS:1;jenkins-hbase20:41937] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-13 06:42:26,850 INFO [RS:1;jenkins-hbase20:41937] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:41937 2023-07-13 06:42:26,859 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:41937-0x1008510fe3f0002, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,41937,1689230543130 2023-07-13 06:42:26,860 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,41937,1689230543130 2023-07-13 06:42:26,860 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 06:42:26,860 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 06:42:26,860 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:41937-0x1008510fe3f0002, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 06:42:26,861 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,41937,1689230543130] 2023-07-13 06:42:26,861 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,41937,1689230543130; numProcessing=1 2023-07-13 06:42:26,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:26,862 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase20.apache.org,41937,1689230543130 znode expired, triggering replicatorRemoved event 2023-07-13 06:42:26,863 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,41937,1689230543130 already deleted, retry=false 2023-07-13 06:42:26,863 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase20.apache.org,41937,1689230543130 on jenkins-hbase20.apache.org,44209,1689230542090 2023-07-13 06:42:26,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:26,870 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase20.apache.org,41937,1689230543130, splitWal=true, meta=false 2023-07-13 06:42:26,870 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase20.apache.org,41937,1689230543130 (carryingMeta=false) jenkins-hbase20.apache.org,41937,1689230543130/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@29b54c6b[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-13 06:42:26,873 INFO [PEWorker-3] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,41937,1689230543130, splitWal=true, meta=false 2023-07-13 06:42:26,875 INFO [PEWorker-3] procedure.ServerCrashProcedure(199): jenkins-hbase20.apache.org,41937,1689230543130 had 1 regions 2023-07-13 06:42:26,876 INFO [PEWorker-3] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,41937,1689230543130, splitWal=true, meta=false, isMeta: false 2023-07-13 06:42:26,878 DEBUG [PEWorker-3] master.MasterWalManager(318): Renamed region directory: hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130-splitting 2023-07-13 06:42:26,880 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase20.apache.org,41937,1689230543130 WAL count=1, meta=false 2023-07-13 06:42:26,883 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485}] 2023-07-13 06:42:26,890 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:26,892 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485, worker=jenkins-hbase20.apache.org,39919,1689230543088}] 2023-07-13 06:42:26,962 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:41937-0x1008510fe3f0002, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-13 06:42:26,962 INFO [RS:1;jenkins-hbase20:41937] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,41937,1689230543130; zookeeper connection closed. 2023-07-13 06:42:26,962 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:41937-0x1008510fe3f0002, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-13 06:42:26,963 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6bf50263] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6bf50263 2023-07-13 06:42:26,972 WARN [RPCClient-NioEventLoopGroup-5-5] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:41937 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:41937 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-13 06:42:26,973 DEBUG [RPCClient-NioEventLoopGroup-5-5] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:41937 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:41937 2023-07-13 06:42:27,057 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39919] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-13 06:42:27,078 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130-splitting/jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485, size=1.1 K (1149bytes) 2023-07-13 06:42:27,079 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130-splitting/jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485 2023-07-13 06:42:27,084 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130-splitting/jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485 after 5ms 2023-07-13 06:42:27,090 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(311): Open hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130-splitting/jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485 took 13ms 2023-07-13 06:42:27,103 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(346): Last flushed sequenceid for fc3ebf4b2149ca5230c85524c0de4bba: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-13 06:42:27,105 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130-splitting/jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485 so closing down 2023-07-13 06:42:27,106 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-13 06:42:27,113 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-13 06:42:27,116 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485.temp 2023-07-13 06:42:27,140 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-13 06:42:27,155 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485.temp (wrote 10 edits, skipped 0 edits in 2 ms) 2023-07-13 06:42:27,157 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485.temp to hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/recovered.edits/0000000000000000013 2023-07-13 06:42:27,158 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 67 ms; skipped=1; WAL=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130-splitting/jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485, size=1.1 K, length=1149, corrupted=false, cancelled=false 2023-07-13 06:42:27,158 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130-splitting/jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485, journal: Splitting hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130-splitting/jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485, size=1.1 K (1149bytes) at 1689230547078Finishing writing output for hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130-splitting/jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485 so closing down at 1689230547106 (+28 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485.temp at 1689230547116 (+10 ms)3 split writer threads finished at 1689230547141 (+25 ms)Closed recovered edits writer path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485.temp (wrote 10 edits, skipped 0 edits in 2 ms) at 1689230547155 (+14 ms)Rename recovered edits hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485.temp to hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/recovered.edits/0000000000000000013 at 1689230547157 (+2 ms)Processed 11 edits across 1 Regions in 67 ms; skipped=1; WAL=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130-splitting/jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485, size=1.1 K, length=1149, corrupted=false, cancelled=false at 1689230547158 (+1 ms) 2023-07-13 06:42:27,159 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130-splitting/jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485 2023-07-13 06:42:27,161 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-13 06:42:27,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44209] master.HMaster(3641): Remote procedure done, pid=14 2023-07-13 06:42:27,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44209] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130-splitting/jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485 to hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/oldWALs 2023-07-13 06:42:27,174 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-13 06:42:27,174 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485, worker=jenkins-hbase20.apache.org,39919,1689230543088 in 279 msec 2023-07-13 06:42:27,176 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:27,180 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-13 06:42:27,181 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase20.apache.org%2C41937%2C1689230543130.1689230544485, worker=jenkins-hbase20.apache.org,39919,1689230543088 in 295 msec 2023-07-13 06:42:27,183 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130-splitting dir is empty, no logs to split. 2023-07-13 06:42:27,184 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase20.apache.org,41937,1689230543130 WAL count=0, meta=false 2023-07-13 06:42:27,184 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase20.apache.org,41937,1689230543130 WAL splitting is done? wals=0, meta=false 2023-07-13 06:42:27,187 INFO [PEWorker-5] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase20.apache.org,41937,1689230543130 failed, ignore...File hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,41937,1689230543130-splitting does not exist. 2023-07-13 06:42:27,189 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=fc3ebf4b2149ca5230c85524c0de4bba, ASSIGN}] 2023-07-13 06:42:27,190 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=fc3ebf4b2149ca5230c85524c0de4bba, ASSIGN 2023-07-13 06:42:27,191 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=fc3ebf4b2149ca5230c85524c0de4bba, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-13 06:42:27,282 DEBUG [hconnection-0x69aa34d9-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:41937 this server is in the failed servers list 2023-07-13 06:42:27,335 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-13 06:42:27,342 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=fc3ebf4b2149ca5230c85524c0de4bba, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:27,342 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689230547342"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689230547342"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689230547342"}]},"ts":"1689230547342"} 2023-07-13 06:42:27,346 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure fc3ebf4b2149ca5230c85524c0de4bba, server=jenkins-hbase20.apache.org,39919,1689230543088}] 2023-07-13 06:42:27,508 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:27,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fc3ebf4b2149ca5230c85524c0de4bba, NAME => 't,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.', STARTKEY => '', ENDKEY => ''} 2023-07-13 06:42:27,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:27,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 06:42:27,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:27,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:27,510 INFO [StoreOpener-fc3ebf4b2149ca5230c85524c0de4bba-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:27,512 DEBUG [StoreOpener-fc3ebf4b2149ca5230c85524c0de4bba-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/f 2023-07-13 06:42:27,512 DEBUG [StoreOpener-fc3ebf4b2149ca5230c85524c0de4bba-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/f 2023-07-13 06:42:27,513 INFO [StoreOpener-fc3ebf4b2149ca5230c85524c0de4bba-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fc3ebf4b2149ca5230c85524c0de4bba columnFamilyName f 2023-07-13 06:42:27,513 INFO [StoreOpener-fc3ebf4b2149ca5230c85524c0de4bba-1] regionserver.HStore(310): Store=fc3ebf4b2149ca5230c85524c0de4bba/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 06:42:27,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:27,519 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:27,521 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/recovered.edits/0000000000000000013 2023-07-13 06:42:27,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/recovered.edits/0000000000000000013 2023-07-13 06:42:27,528 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing fc3ebf4b2149ca5230c85524c0de4bba 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-13 06:42:27,575 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/.tmp/f/9a13780ce1a446e6ae546f0a3c18c2dd 2023-07-13 06:42:27,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/.tmp/f/9a13780ce1a446e6ae546f0a3c18c2dd as hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/f/9a13780ce1a446e6ae546f0a3c18c2dd 2023-07-13 06:42:27,631 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/f/9a13780ce1a446e6ae546f0a3c18c2dd, entries=10, sequenceid=13, filesize=4.8 K 2023-07-13 06:42:27,632 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for fc3ebf4b2149ca5230c85524c0de4bba in 104ms, sequenceid=13, compaction requested=false; wal=null 2023-07-13 06:42:27,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-13 06:42:27,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/recovered.edits/0000000000000000013 2023-07-13 06:42:27,642 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:27,646 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-13 06:42:27,647 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened fc3ebf4b2149ca5230c85524c0de4bba; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60418591, jitterRate=-0.09969283640384674}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-13 06:42:27,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for fc3ebf4b2149ca5230c85524c0de4bba: 2023-07-13 06:42:27,648 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba., pid=16, masterSystemTime=1689230547502 2023-07-13 06:42:27,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:27,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:27,651 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=fc3ebf4b2149ca5230c85524c0de4bba, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:27,652 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689230547651"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689230547651"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689230547651"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689230547651"}]},"ts":"1689230547651"} 2023-07-13 06:42:27,657 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-13 06:42:27,657 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure fc3ebf4b2149ca5230c85524c0de4bba, server=jenkins-hbase20.apache.org,39919,1689230543088 in 308 msec 2023-07-13 06:42:27,659 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-13 06:42:27,659 INFO [PEWorker-3] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase20.apache.org,41937,1689230543130 after splitting done 2023-07-13 06:42:27,659 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=fc3ebf4b2149ca5230c85524c0de4bba, ASSIGN in 468 msec 2023-07-13 06:42:27,659 DEBUG [PEWorker-3] master.DeadServer(114): Removed jenkins-hbase20.apache.org,41937,1689230543130 from processing; numProcessing=0 2023-07-13 06:42:27,662 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase20.apache.org,41937,1689230543130, splitWal=true, meta=false in 796 msec 2023-07-13 06:42:27,814 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39c41235 to 127.0.0.1:58518 2023-07-13 06:42:27,814 DEBUG [Listener at localhost.localdomain/40319] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 06:42:27,814 INFO [Listener at localhost.localdomain/40319] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-13 06:42:27,833 INFO [Listener at localhost.localdomain/40319] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=272 (was 296), OpenFileDescriptor=567 (was 585), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=252 (was 249) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=6213 (was 6281) 2023-07-13 06:42:27,848 INFO [Listener at localhost.localdomain/40319] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=272, OpenFileDescriptor=567, MaxFileDescriptor=60000, SystemLoadAverage=252, ProcessCount=168, AvailableMemoryMB=6213 2023-07-13 06:42:27,848 INFO [Listener at localhost.localdomain/40319] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase20:41937 2023-07-13 06:42:27,862 INFO [Listener at localhost.localdomain/40319] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-13 06:42:27,862 INFO [Listener at localhost.localdomain/40319] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-13 06:42:27,862 INFO [Listener at localhost.localdomain/40319] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-13 06:42:27,862 INFO [Listener at localhost.localdomain/40319] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-13 06:42:27,863 INFO [Listener at localhost.localdomain/40319] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-13 06:42:27,863 INFO [Listener at localhost.localdomain/40319] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-13 06:42:27,863 INFO [Listener at localhost.localdomain/40319] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-13 06:42:27,864 INFO [Listener at localhost.localdomain/40319] ipc.NettyRpcServer(120): Bind to /148.251.75.209:34261 2023-07-13 06:42:27,865 INFO [Listener at localhost.localdomain/40319] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-13 06:42:27,866 DEBUG [Listener at localhost.localdomain/40319] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-13 06:42:27,866 INFO [Listener at localhost.localdomain/40319] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 06:42:27,868 INFO [Listener at localhost.localdomain/40319] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 06:42:27,869 INFO [Listener at localhost.localdomain/40319] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34261 connecting to ZooKeeper ensemble=127.0.0.1:58518 2023-07-13 06:42:27,872 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:342610x0, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-13 06:42:27,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:34261-0x1008510fe3f0008 connected 2023-07-13 06:42:27,873 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ZKUtil(162): regionserver:34261-0x1008510fe3f0008, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-13 06:42:27,874 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ZKUtil(162): regionserver:34261-0x1008510fe3f0008, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-13 06:42:27,875 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ZKUtil(164): regionserver:34261-0x1008510fe3f0008, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-13 06:42:27,876 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34261 2023-07-13 06:42:27,876 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34261 2023-07-13 06:42:27,878 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34261 2023-07-13 06:42:27,879 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34261 2023-07-13 06:42:27,879 DEBUG [Listener at localhost.localdomain/40319] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34261 2023-07-13 06:42:27,881 INFO [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(951): ClusterId : c61308e6-af66-4627-b6a4-80b47ec8901e 2023-07-13 06:42:27,882 DEBUG [RS:2;jenkins-hbase20:34261] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-13 06:42:27,888 DEBUG [RS:2;jenkins-hbase20:34261] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-13 06:42:27,888 DEBUG [RS:2;jenkins-hbase20:34261] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-13 06:42:27,894 DEBUG [RS:2;jenkins-hbase20:34261] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-13 06:42:27,895 DEBUG [RS:2;jenkins-hbase20:34261] zookeeper.ReadOnlyZKClient(139): Connect 0x6a0e5441 to 127.0.0.1:58518 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-13 06:42:27,904 DEBUG [RS:2;jenkins-hbase20:34261] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64c9b770, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-13 06:42:27,905 DEBUG [RS:2;jenkins-hbase20:34261] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@60f63f31, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-13 06:42:27,914 DEBUG [RS:2;jenkins-hbase20:34261] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase20:34261 2023-07-13 06:42:27,914 INFO [RS:2;jenkins-hbase20:34261] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-13 06:42:27,914 INFO [RS:2;jenkins-hbase20:34261] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-13 06:42:27,914 DEBUG [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(1022): About to register with Master. 2023-07-13 06:42:27,915 INFO [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,44209,1689230542090 with isa=jenkins-hbase20.apache.org/148.251.75.209:34261, startcode=1689230547862 2023-07-13 06:42:27,915 DEBUG [RS:2;jenkins-hbase20:34261] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-13 06:42:27,920 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:32909, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-13 06:42:27,920 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44209] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:27,921 DEBUG [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c 2023-07-13 06:42:27,921 DEBUG [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:46147 2023-07-13 06:42:27,921 DEBUG [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-13 06:42:27,922 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 06:42:27,922 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 06:42:27,923 DEBUG [RS:2;jenkins-hbase20:34261] zookeeper.ZKUtil(162): regionserver:34261-0x1008510fe3f0008, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:27,923 WARN [RS:2;jenkins-hbase20:34261] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-13 06:42:27,923 INFO [RS:2;jenkins-hbase20:34261] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-13 06:42:27,924 DEBUG [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:27,925 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,34261,1689230547862] 2023-07-13 06:42:27,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:27,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:27,933 DEBUG [RS:2;jenkins-hbase20:34261] zookeeper.ZKUtil(162): regionserver:34261-0x1008510fe3f0008, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:27,933 DEBUG [RS:2;jenkins-hbase20:34261] zookeeper.ZKUtil(162): regionserver:34261-0x1008510fe3f0008, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:27,934 DEBUG [RS:2;jenkins-hbase20:34261] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-13 06:42:27,935 INFO [RS:2;jenkins-hbase20:34261] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-13 06:42:27,937 INFO [RS:2;jenkins-hbase20:34261] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-13 06:42:27,939 INFO [RS:2;jenkins-hbase20:34261] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-13 06:42:27,939 INFO [RS:2;jenkins-hbase20:34261] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:27,942 INFO [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-13 06:42:27,944 INFO [RS:2;jenkins-hbase20:34261] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:27,945 DEBUG [RS:2;jenkins-hbase20:34261] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:27,945 DEBUG [RS:2;jenkins-hbase20:34261] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:27,945 DEBUG [RS:2;jenkins-hbase20:34261] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:27,945 DEBUG [RS:2;jenkins-hbase20:34261] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:27,945 DEBUG [RS:2;jenkins-hbase20:34261] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:27,945 DEBUG [RS:2;jenkins-hbase20:34261] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-13 06:42:27,945 DEBUG [RS:2;jenkins-hbase20:34261] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:27,945 DEBUG [RS:2;jenkins-hbase20:34261] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:27,945 DEBUG [RS:2;jenkins-hbase20:34261] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:27,945 DEBUG [RS:2;jenkins-hbase20:34261] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-13 06:42:27,946 INFO [RS:2;jenkins-hbase20:34261] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:27,947 INFO [RS:2;jenkins-hbase20:34261] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:27,947 INFO [RS:2;jenkins-hbase20:34261] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:27,960 INFO [RS:2;jenkins-hbase20:34261] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-13 06:42:27,960 INFO [RS:2;jenkins-hbase20:34261] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34261,1689230547862-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-13 06:42:27,971 INFO [RS:2;jenkins-hbase20:34261] regionserver.Replication(203): jenkins-hbase20.apache.org,34261,1689230547862 started 2023-07-13 06:42:27,971 INFO [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,34261,1689230547862, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:34261, sessionid=0x1008510fe3f0008 2023-07-13 06:42:27,971 INFO [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase20:34261,5,FailOnTimeoutGroup] 2023-07-13 06:42:27,971 DEBUG [RS:2;jenkins-hbase20:34261] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-13 06:42:27,972 DEBUG [RS:2;jenkins-hbase20:34261] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:27,972 DEBUG [RS:2;jenkins-hbase20:34261] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,34261,1689230547862' 2023-07-13 06:42:27,972 DEBUG [RS:2;jenkins-hbase20:34261] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-13 06:42:27,972 INFO [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase20.apache.org,41937,1689230543130 2023-07-13 06:42:27,972 INFO [Listener at localhost.localdomain/40319] client.TestScannerTimeout(177): START ************ test3686b 2023-07-13 06:42:27,972 DEBUG [RS:2;jenkins-hbase20:34261] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-13 06:42:27,973 DEBUG [RS:2;jenkins-hbase20:34261] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-13 06:42:27,973 DEBUG [RS:2;jenkins-hbase20:34261] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-13 06:42:27,973 DEBUG [RS:2;jenkins-hbase20:34261] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:27,973 DEBUG [RS:2;jenkins-hbase20:34261] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,34261,1689230547862' 2023-07-13 06:42:27,973 DEBUG [RS:2;jenkins-hbase20:34261] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-13 06:42:27,973 DEBUG [RS:2;jenkins-hbase20:34261] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-13 06:42:27,974 DEBUG [RS:2;jenkins-hbase20:34261] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-13 06:42:27,974 INFO [RS:2;jenkins-hbase20:34261] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-13 06:42:27,974 INFO [RS:2;jenkins-hbase20:34261] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-13 06:42:27,975 DEBUG [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-13 06:42:27,975 DEBUG [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:27,975 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ReadOnlyZKClient(139): Connect 0x7a51b280 to 127.0.0.1:58518 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-13 06:42:27,979 DEBUG [Listener at localhost.localdomain/40319] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@690525ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-13 06:42:27,982 DEBUG [hconnection-0x4904fe8b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 06:42:27,985 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47940, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 06:42:28,080 INFO [RS:2;jenkins-hbase20:34261] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34261%2C1689230547862, suffix=, logDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,34261,1689230547862, archiveDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/oldWALs, maxLogs=32 2023-07-13 06:42:28,102 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46221,DS-9f07bcbe-b9b6-483b-b54a-1fd62e8b70d4,DISK] 2023-07-13 06:42:28,103 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32857,DS-bd5046f7-7437-4b40-ab6d-2b891ce9bd15,DISK] 2023-07-13 06:42:28,106 INFO [RS:2;jenkins-hbase20:34261] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,34261,1689230547862/jenkins-hbase20.apache.org%2C34261%2C1689230547862.1689230548084 2023-07-13 06:42:28,106 DEBUG [RS:2;jenkins-hbase20:34261] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46221,DS-9f07bcbe-b9b6-483b-b54a-1fd62e8b70d4,DISK], DatanodeInfoWithStorage[127.0.0.1:32857,DS-bd5046f7-7437-4b40-ab6d-2b891ce9bd15,DISK]] 2023-07-13 06:42:30,326 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-13 06:42:30,391 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-13 06:42:30,391 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-13 06:42:32,214 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-13 06:42:32,216 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-13 06:42:42,991 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease -1929413452414058477 expired clientIPAndPort=148.251.75.209:47940, userName=jenkins, regionInfo=t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:44,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39919] regionserver.RSRpcServices(3094): Client tried to access missing scanner -1929413452414058477 2023-07-13 06:42:44,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39919] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 148.251.75.209:47940 deadline: 1689230624992, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-1929413452414058477'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-13 06:42:44,994 DEBUG [Listener at localhost.localdomain/40319] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-1929413452414058477'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-1929413452414058477'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-1929413452414058477'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-13 06:42:45,000 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a51b280 to 127.0.0.1:58518 2023-07-13 06:42:45,000 DEBUG [Listener at localhost.localdomain/40319] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 06:42:45,000 INFO [Listener at localhost.localdomain/40319] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-13 06:42:45,019 INFO [Listener at localhost.localdomain/40319] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=295 (was 272) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/cluster_8d5ebd29-c8d7-9901-4c53-f6c110004b44/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58518@0x6a0e5441 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/130930241.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34261 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase20:34261-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=34261 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34261 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-578202921-148.251.75.209-1689230539315:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: jenkins-hbase20:34261Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=34261 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=34261 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c-prefix:jenkins-hbase20.apache.org,34261,1689230547862 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58518@0x6a0e5441-SendThread(127.0.0.1:58518) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58518@0x7a51b280-SendThread(127.0.0.1:58518) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/cluster_8d5ebd29-c8d7-9901-4c53-f6c110004b44/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58518@0x6a0e5441-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34261 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase20:34261 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34261 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-578202921-148.251.75.209-1689230539315:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1562064403_17 at /127.0.0.1:51958 [Receiving block BP-578202921-148.251.75.209-1689230539315:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34261 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost.localdomain:46147 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58518@0x7a51b280 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/130930241.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58518@0x7a51b280-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1562064403_17 at /127.0.0.1:38052 [Receiving block BP-578202921-148.251.75.209-1689230539315:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34261 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=34261 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) - Thread LEAK? -, OpenFileDescriptor=571 (was 567) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 252), ProcessCount=168 (was 168), AvailableMemoryMB=6142 (was 6213) 2023-07-13 06:42:45,033 INFO [Listener at localhost.localdomain/40319] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=295, OpenFileDescriptor=571, MaxFileDescriptor=60000, SystemLoadAverage=204, ProcessCount=168, AvailableMemoryMB=6141 2023-07-13 06:42:45,034 INFO [Listener at localhost.localdomain/40319] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase20:41937 2023-07-13 06:42:45,034 INFO [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase20.apache.org,41937,1689230543130 2023-07-13 06:42:45,034 INFO [Listener at localhost.localdomain/40319] client.TestScannerTimeout(104): START************ test2772 2023-07-13 06:42:45,038 DEBUG [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-13 06:42:45,038 DEBUG [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:45,039 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ReadOnlyZKClient(139): Connect 0x0539e0f9 to 127.0.0.1:58518 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-13 06:42:45,043 DEBUG [Listener at localhost.localdomain/40319] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b53b7ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-13 06:42:45,043 ERROR [Listener at localhost.localdomain/40319] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,39919,1689230543088: die! ***** 2023-07-13 06:42:45,043 ERROR [Listener at localhost.localdomain/40319] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-13 06:42:45,044 DEBUG [Listener at localhost.localdomain/40319] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-13 06:42:45,044 DEBUG [Listener at localhost.localdomain/40319] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-13 06:42:45,050 DEBUG [Listener at localhost.localdomain/40319] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-13 06:42:45,052 DEBUG [Listener at localhost.localdomain/40319] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-13 06:42:45,073 INFO [Listener at localhost.localdomain/40319] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1009778688, "init": 524288000, "max": 2051014656, "used": 247447264 }, "NonHeapMemoryUsage": { "committed": 122249216, "init": 2555904, "max": -1, "used": 119779288 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 3, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 11986, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 61, "ProcessCallTime_min": 1, "ProcessCallTime_max": 2, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 2, "ProcessCallTime_75th_percentile": 2, "ProcessCallTime_90th_percentile": 2, "ProcessCallTime_95th_percentile": 2, "ProcessCallTime_98th_percentile": 2, "ProcessCallTime_99th_percentile": 2, "ProcessCallTime_99.9th_percentile": 2, "ProcessCallTime_TimeRangeCount_0-1": 1, "ProcessCallTime_TimeRangeCount_1-3": 1, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 61, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 61, "TotalCallTime_min": 1, "TotalCallTime_max": 2, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 2, "TotalCallTime_75th_percentile": 2, "TotalCallTime_90th_percentile": 2, "TotalCallTime_95th_percentile": 2, "TotalCallTime_98th_percentile": 2, "TotalCallTime_99th_percentile": 2, "TotalCallTime_99.9th_percentile": 2, "TotalCallTime_TimeRangeCount_0-1": 1, "TotalCallTime_TimeRangeCount_1-3": 1, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 61, "ResponseSize_min": 197, "ResponseSize_max": 681, "ResponseSize_mean": 439, "ResponseSize_25th_percentile": 197, "ResponseSize_median": 197, "ResponseSize_75th_percentile": 678, "ResponseSize_90th_percentile": 679, "ResponseSize_95th_percentile": 680, "ResponseSize_98th_percentile": 680, "ResponseSize_99th_percentile": 680, "ResponseSize_99.9th_percentile": 680, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 4, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 61, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16719 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:58518", "tag.serverName": "jenkins-hbase20.apache.org,41937,1689230543130", "tag.clusterId": "c61308e6-af66-4627-b6a4-80b47ec8901e", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "regionCount": 0, "storeCount": 0, "hlogFileCount": 1, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 0, "avgStoreFileAge": 0, "numReferenceFiles": 0, "regionServerStartTime": 1689230543130, "averageRegionSize": 0, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 546, "localBytesRead": 546, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 1640, "ByteBuffAllocatorPoolAllocationBytes": 66560, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 11, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 0, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 1, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 10, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 1, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 10, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 0, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 0, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 0, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 0, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 1, "ScanSize_min": 0, "ScanSize_max": 0, "ScanSize_mean": 0, "ScanSize_25th_percentile": 0, "ScanSize_median": 0, "ScanSize_75th_percentile": 0, "ScanSize_90th_percentile": 0, "ScanSize_95th_percentile": 0, "ScanSize_98th_percentile": 0, "ScanSize_99th_percentile": 0, "ScanSize_99.9th_percentile": 0, "slowGetCount": 0, "flushedMemstoreBytes": 0, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 0, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.6024676104956077, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 24, "ServerWriteQueryPerSecond_mean_rate": 1.156651993728719, "ServerWriteQueryPerSecond_1min_rate": 3.738243758742744, "ServerWriteQueryPerSecond_5min_rate": 4.565901237603426, "ServerWriteQueryPerSecond_15min_rate": 4.720662978343764 } ] } 2023-07-13 06:42:45,075 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=44209] master.MasterRpcServices(612): jenkins-hbase20.apache.org,39919,1689230543088 reported a fatal error: ***** ABORTING region server jenkins-hbase20.apache.org,39919,1689230543088: die! ***** 2023-07-13 06:42:45,076 INFO [Listener at localhost.localdomain/40319] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,39919,1689230543088' ***** 2023-07-13 06:42:45,076 INFO [Listener at localhost.localdomain/40319] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-13 06:42:45,077 INFO [RS:0;jenkins-hbase20:39919] regionserver.HeapMemoryManager(220): Stopping 2023-07-13 06:42:45,077 INFO [RS:0;jenkins-hbase20:39919] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-13 06:42:45,077 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-13 06:42:45,077 INFO [RS:0;jenkins-hbase20:39919] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-13 06:42:45,077 INFO [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(3305): Received CLOSE for a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:45,077 INFO [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(3305): Received CLOSE for fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:45,077 INFO [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:45,078 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing a76e4d3d13ae5bb10f9d5a76e91300ce, disabling compactions & flushes 2023-07-13 06:42:45,078 DEBUG [RS:0;jenkins-hbase20:39919] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3954eb73 to 127.0.0.1:58518 2023-07-13 06:42:45,078 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:45,078 DEBUG [RS:0;jenkins-hbase20:39919] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 06:42:45,078 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:45,078 INFO [RS:0;jenkins-hbase20:39919] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-13 06:42:45,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. after waiting 0 ms 2023-07-13 06:42:45,079 INFO [RS:0;jenkins-hbase20:39919] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-13 06:42:45,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:45,079 INFO [RS:0;jenkins-hbase20:39919] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-13 06:42:45,079 INFO [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-13 06:42:45,082 INFO [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-13 06:42:45,083 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:45,083 DEBUG [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(1478): Online Regions={a76e4d3d13ae5bb10f9d5a76e91300ce=hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce., 1588230740=hbase:meta,,1.1588230740, fc3ebf4b2149ca5230c85524c0de4bba=t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.} 2023-07-13 06:42:45,083 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-13 06:42:45,084 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:45,084 DEBUG [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(1504): Waiting on 1588230740, a76e4d3d13ae5bb10f9d5a76e91300ce, fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:45,084 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-13 06:42:45,085 DEBUG [hconnection-0xad6b320-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 06:42:45,084 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for a76e4d3d13ae5bb10f9d5a76e91300ce: 2023-07-13 06:42:45,085 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-13 06:42:45,086 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-13 06:42:45,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:45,086 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-13 06:42:45,087 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing fc3ebf4b2149ca5230c85524c0de4bba, disabling compactions & flushes 2023-07-13 06:42:45,087 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:45,087 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:45,087 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 3234 in region hbase:meta,,1.1588230740 2023-07-13 06:42:45,087 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. after waiting 0 ms 2023-07-13 06:42:45,087 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-13 06:42:45,087 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:45,089 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-13 06:42:45,089 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-13 06:42:45,089 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-13 06:42:45,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:45,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for fc3ebf4b2149ca5230c85524c0de4bba: 2023-07-13 06:42:45,091 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54646, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 06:42:45,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:45,092 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39919] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,39919,1689230543088 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-13 06:42:45,092 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39919] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 148.251.75.209:54646 deadline: 1689230625091, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,39919,1689230543088 aborting 2023-07-13 06:42:45,198 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39919] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,39919,1689230543088 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-13 06:42:45,198 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39919] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 148.251.75.209:54646 deadline: 1689230625198, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,39919,1689230543088 aborting 2023-07-13 06:42:45,285 INFO [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,39919,1689230543088; all regions closed. 2023-07-13 06:42:45,306 DEBUG [RS:0;jenkins-hbase20:39919] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 06:42:45,306 INFO [RS:0;jenkins-hbase20:39919] regionserver.LeaseManager(133): Closed leases 2023-07-13 06:42:45,306 INFO [RS:0;jenkins-hbase20:39919] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-13 06:42:45,307 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-13 06:42:45,308 INFO [RS:0;jenkins-hbase20:39919] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:39919 2023-07-13 06:42:45,311 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:45,311 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:34261-0x1008510fe3f0008, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,39919,1689230543088 2023-07-13 06:42:45,311 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 06:42:45,311 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:34261-0x1008510fe3f0008, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 06:42:45,311 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 06:42:45,312 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,39919,1689230543088] 2023-07-13 06:42:45,312 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,39919,1689230543088; numProcessing=1 2023-07-13 06:42:45,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34261-0x1008510fe3f0008, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:45,313 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase20.apache.org,39919,1689230543088 znode expired, triggering replicatorRemoved event 2023-07-13 06:42:45,313 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,39919,1689230543088 already deleted, retry=false 2023-07-13 06:42:45,313 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase20.apache.org,39919,1689230543088 on jenkins-hbase20.apache.org,44209,1689230542090 2023-07-13 06:42:45,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34261-0x1008510fe3f0008, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:45,314 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=17, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase20.apache.org,39919,1689230543088, splitWal=true, meta=true 2023-07-13 06:42:45,314 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=17 for jenkins-hbase20.apache.org,39919,1689230543088 (carryingMeta=true) jenkins-hbase20.apache.org,39919,1689230543088/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@38bfe208[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-13 06:42:45,316 INFO [PEWorker-2] procedure.ServerCrashProcedure(161): Start pid=17, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,39919,1689230543088, splitWal=true, meta=true 2023-07-13 06:42:45,317 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,39919,1689230543088, splitWal=true, meta=true, isMeta: true 2023-07-13 06:42:45,322 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting 2023-07-13 06:42:45,323 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase20.apache.org,39919,1689230543088 WAL count=1, meta=true 2023-07-13 06:42:45,324 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta}] 2023-07-13 06:42:45,326 DEBUG [PEWorker-2] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:45,327 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta, worker=jenkins-hbase20.apache.org,34261,1689230547862}] 2023-07-13 06:42:45,402 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:39919 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:39919 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-13 06:42:45,403 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:39919 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:39919 2023-07-13 06:42:45,412 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-13 06:42:45,412 INFO [RS:0;jenkins-hbase20:39919] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,39919,1689230543088; zookeeper connection closed. 2023-07-13 06:42:45,412 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:39919-0x1008510fe3f0001, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-13 06:42:45,412 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@702484f] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@702484f 2023-07-13 06:42:45,480 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:45,481 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-13 06:42:45,483 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57380, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-13 06:42:45,483 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34261] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=19 2023-07-13 06:42:45,499 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta, size=4.0 K (4130bytes) 2023-07-13 06:42:45,499 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta 2023-07-13 06:42:45,501 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta after 2ms 2023-07-13 06:42:45,505 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(311): Open hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta took 6ms 2023-07-13 06:42:45,510 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-13 06:42:45,512 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta so closing down 2023-07-13 06:42:45,512 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-13 06:42:45,515 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta.temp 2023-07-13 06:42:45,516 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-13 06:42:45,529 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) 2023-07-13 06:42:45,531 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta.temp to hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-13 06:42:45,531 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(423): Processed 13 edits across 1 Regions in 26 ms; skipped=1; WAL=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta, size=4.0 K, length=4130, corrupted=false, cancelled=false 2023-07-13 06:42:45,532 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta, journal: Splitting hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta, size=4.0 K (4130bytes) at 1689230565499Finishing writing output for hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta so closing down at 1689230565512 (+13 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta.temp at 1689230565515 (+3 ms)3 split writer threads finished at 1689230565516 (+1 ms)Closed recovered edits writer path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) at 1689230565529 (+13 ms)Rename recovered edits hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta.temp to hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 at 1689230565531 (+2 ms)Processed 13 edits across 1 Regions in 26 ms; skipped=1; WAL=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta, size=4.0 K, length=4130, corrupted=false, cancelled=false at 1689230565531 2023-07-13 06:42:45,532 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta 2023-07-13 06:42:45,532 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=19 2023-07-13 06:42:45,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44209] master.HMaster(3641): Remote procedure done, pid=19 2023-07-13 06:42:45,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44209] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta to hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/oldWALs 2023-07-13 06:42:45,541 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=18 2023-07-13 06:42:45,542 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta, worker=jenkins-hbase20.apache.org,34261,1689230547862 in 211 msec 2023-07-13 06:42:45,543 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:45,546 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-13 06:42:45,547 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase20.apache.org%2C39919%2C1689230543088.meta.1689230544720.meta, worker=jenkins-hbase20.apache.org,34261,1689230547862 in 220 msec 2023-07-13 06:42:45,549 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting dir is empty, no logs to split. 2023-07-13 06:42:45,549 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase20.apache.org,39919,1689230543088 WAL count=0, meta=true 2023-07-13 06:42:45,549 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase20.apache.org,39919,1689230543088 WAL splitting is done? wals=0, meta=true 2023-07-13 06:42:45,550 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-13 06:42:45,551 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-13 06:42:45,552 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-13 06:42:45,604 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-13 06:42:45,702 DEBUG [jenkins-hbase20:44209] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-13 06:42:45,703 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,34261,1689230547862, state=OPENING 2023-07-13 06:42:45,705 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-13 06:42:45,706 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,34261,1689230547862}] 2023-07-13 06:42:45,706 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-13 06:42:45,708 WARN [ReadOnlyZKClient-127.0.0.1:58518@0x0539e0f9] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-13 06:42:45,709 DEBUG [hconnection-0xad6b320-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 06:42:45,716 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57388, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 06:42:45,717 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34261] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 111 connection: 148.251.75.209:57388 deadline: 1689230625716, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:45,877 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-13 06:42:45,877 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-13 06:42:45,879 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34261%2C1689230547862.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,34261,1689230547862, archiveDir=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/oldWALs, maxLogs=32 2023-07-13 06:42:45,895 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46221,DS-9f07bcbe-b9b6-483b-b54a-1fd62e8b70d4,DISK] 2023-07-13 06:42:45,895 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32857,DS-bd5046f7-7437-4b40-ab6d-2b891ce9bd15,DISK] 2023-07-13 06:42:45,898 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,34261,1689230547862/jenkins-hbase20.apache.org%2C34261%2C1689230547862.meta.1689230565880.meta 2023-07-13 06:42:45,898 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46221,DS-9f07bcbe-b9b6-483b-b54a-1fd62e8b70d4,DISK], DatanodeInfoWithStorage[127.0.0.1:32857,DS-bd5046f7-7437-4b40-ab6d-2b891ce9bd15,DISK]] 2023-07-13 06:42:45,899 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-13 06:42:45,899 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-13 06:42:45,899 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-13 06:42:45,900 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-13 06:42:45,900 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-13 06:42:45,900 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 06:42:45,900 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-13 06:42:45,900 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-13 06:42:45,902 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-13 06:42:45,904 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/info 2023-07-13 06:42:45,905 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/info 2023-07-13 06:42:45,905 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-13 06:42:45,906 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 06:42:45,906 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-13 06:42:45,908 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/rep_barrier 2023-07-13 06:42:45,908 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/rep_barrier 2023-07-13 06:42:45,909 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-13 06:42:45,909 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 06:42:45,909 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-13 06:42:45,911 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/table 2023-07-13 06:42:45,911 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/table 2023-07-13 06:42:45,911 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-13 06:42:45,913 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 06:42:45,914 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740 2023-07-13 06:42:45,917 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740 2023-07-13 06:42:45,918 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-13 06:42:45,924 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5464): Applied 32, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=15, path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-13 06:42:45,925 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=3.16 KB heapSize=6.27 KB 2023-07-13 06:42:45,943 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.00 KB at sequenceid=15 (bloomFilter=false), to=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/.tmp/info/0b58caa71b3747728c821e53ca738dff 2023-07-13 06:42:45,971 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=15 (bloomFilter=false), to=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/.tmp/table/05b23a81877a41bdba98d4233ed9d99c 2023-07-13 06:42:45,981 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/.tmp/info/0b58caa71b3747728c821e53ca738dff as hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/info/0b58caa71b3747728c821e53ca738dff 2023-07-13 06:42:45,990 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/info/0b58caa71b3747728c821e53ca738dff, entries=28, sequenceid=15, filesize=7.7 K 2023-07-13 06:42:45,992 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/.tmp/table/05b23a81877a41bdba98d4233ed9d99c as hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/table/05b23a81877a41bdba98d4233ed9d99c 2023-07-13 06:42:46,000 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/table/05b23a81877a41bdba98d4233ed9d99c, entries=4, sequenceid=15, filesize=4.7 K 2023-07-13 06:42:46,000 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.16 KB/3234, heapSize ~5.98 KB/6128, currentSize=0 B/0 for 1588230740 in 75ms, sequenceid=15, compaction requested=false; wal=null 2023-07-13 06:42:46,001 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-13 06:42:46,005 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-13 06:42:46,007 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-13 06:42:46,011 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=1 2023-07-13 06:42:46,013 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68128945, jitterRate=0.015200391411781311}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-13 06:42:46,013 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-13 06:42:46,014 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=21, masterSystemTime=1689230565865 2023-07-13 06:42:46,016 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-13 06:42:46,017 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-13 06:42:46,018 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,34261,1689230547862, state=OPEN 2023-07-13 06:42:46,019 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-13 06:42:46,019 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-13 06:42:46,021 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-13 06:42:46,021 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,34261,1689230547862 in 313 msec 2023-07-13 06:42:46,024 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=17 2023-07-13 06:42:46,024 INFO [PEWorker-2] procedure.ServerCrashProcedure(199): jenkins-hbase20.apache.org,39919,1689230543088 had 3 regions 2023-07-13 06:42:46,024 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 471 msec 2023-07-13 06:42:46,025 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,39919,1689230543088, splitWal=true, meta=true, isMeta: false 2023-07-13 06:42:46,027 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase20.apache.org,39919,1689230543088 WAL count=1, meta=false 2023-07-13 06:42:46,028 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485}] 2023-07-13 06:42:46,030 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:46,031 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485, worker=jenkins-hbase20.apache.org,34261,1689230547862}] 2023-07-13 06:42:46,184 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34261] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=23 2023-07-13 06:42:46,197 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485, size=901 (901bytes) 2023-07-13 06:42:46,197 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485 2023-07-13 06:42:46,197 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485 after 0ms 2023-07-13 06:42:46,200 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(311): Open hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485 took 4ms 2023-07-13 06:42:46,203 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(346): Last flushed sequenceid for a76e4d3d13ae5bb10f9d5a76e91300ce: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-13 06:42:46,206 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(346): Last flushed sequenceid for fc3ebf4b2149ca5230c85524c0de4bba: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-13 06:42:46,206 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485 so closing down 2023-07-13 06:42:46,206 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-13 06:42:46,208 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485.temp 2023-07-13 06:42:46,208 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-13 06:42:46,219 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-13 06:42:46,221 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485.temp to hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/recovered.edits/0000000000000000005 2023-07-13 06:42:46,221 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(423): Processed 4 edits across 1 Regions in 20 ms; skipped=2; WAL=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485, size=901, length=901, corrupted=false, cancelled=false 2023-07-13 06:42:46,221 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485, journal: Splitting hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485, size=901 (901bytes) at 1689230566197Finishing writing output for hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485 so closing down at 1689230566206 (+9 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485.temp at 1689230566208 (+2 ms)3 split writer threads finished at 1689230566209 (+1 ms)Closed recovered edits writer path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689230566219 (+10 ms)Rename recovered edits hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485.temp to hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/recovered.edits/0000000000000000005 at 1689230566221 (+2 ms)Processed 4 edits across 1 Regions in 20 ms; skipped=2; WAL=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485, size=901, length=901, corrupted=false, cancelled=false at 1689230566221 2023-07-13 06:42:46,221 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485 2023-07-13 06:42:46,221 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=23 2023-07-13 06:42:46,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44209] master.HMaster(3641): Remote procedure done, pid=23 2023-07-13 06:42:46,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44209] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting/jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485 to hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/oldWALs 2023-07-13 06:42:46,230 DEBUG [hconnection-0xad6b320-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:39919 this server is in the failed servers list 2023-07-13 06:42:46,230 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-13 06:42:46,230 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485, worker=jenkins-hbase20.apache.org,34261,1689230547862 in 196 msec 2023-07-13 06:42:46,232 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:46,234 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=17 2023-07-13 06:42:46,234 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase20.apache.org%2C39919%2C1689230543088.1689230544485, worker=jenkins-hbase20.apache.org,34261,1689230547862 in 205 msec 2023-07-13 06:42:46,236 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/WALs/jenkins-hbase20.apache.org,39919,1689230543088-splitting dir is empty, no logs to split. 2023-07-13 06:42:46,236 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase20.apache.org,39919,1689230543088 WAL count=0, meta=false 2023-07-13 06:42:46,236 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase20.apache.org,39919,1689230543088 WAL splitting is done? wals=0, meta=false 2023-07-13 06:42:46,238 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a76e4d3d13ae5bb10f9d5a76e91300ce, ASSIGN}, {pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=fc3ebf4b2149ca5230c85524c0de4bba, ASSIGN}] 2023-07-13 06:42:46,239 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=fc3ebf4b2149ca5230c85524c0de4bba, ASSIGN 2023-07-13 06:42:46,239 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a76e4d3d13ae5bb10f9d5a76e91300ce, ASSIGN 2023-07-13 06:42:46,240 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=a76e4d3d13ae5bb10f9d5a76e91300ce, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-13 06:42:46,240 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=fc3ebf4b2149ca5230c85524c0de4bba, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-13 06:42:46,344 DEBUG [hconnection-0xad6b320-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:39919 this server is in the failed servers list 2023-07-13 06:42:46,391 DEBUG [jenkins-hbase20:44209] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-13 06:42:46,391 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=fc3ebf4b2149ca5230c85524c0de4bba, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:46,391 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=a76e4d3d13ae5bb10f9d5a76e91300ce, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:46,391 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689230566391"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689230566391"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689230566391"}]},"ts":"1689230566391"} 2023-07-13 06:42:46,392 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689230566391"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689230566391"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689230566391"}]},"ts":"1689230566391"} 2023-07-13 06:42:46,392 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:39919 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:39919 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-13 06:42:46,394 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:39919 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:39919 2023-07-13 06:42:46,500 DEBUG [PEWorker-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 06:42:46,504 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57400, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 06:42:46,507 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; OpenRegionProcedure fc3ebf4b2149ca5230c85524c0de4bba, server=jenkins-hbase20.apache.org,34261,1689230547862}] 2023-07-13 06:42:46,508 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=24, state=RUNNABLE; OpenRegionProcedure a76e4d3d13ae5bb10f9d5a76e91300ce, server=jenkins-hbase20.apache.org,34261,1689230547862}] 2023-07-13 06:42:46,554 DEBUG [hconnection-0xad6b320-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:39919 this server is in the failed servers list 2023-07-13 06:42:46,667 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:46,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fc3ebf4b2149ca5230c85524c0de4bba, NAME => 't,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.', STARTKEY => '', ENDKEY => ''} 2023-07-13 06:42:46,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:46,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 06:42:46,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:46,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:46,671 INFO [StoreOpener-fc3ebf4b2149ca5230c85524c0de4bba-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:46,673 DEBUG [StoreOpener-fc3ebf4b2149ca5230c85524c0de4bba-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/f 2023-07-13 06:42:46,673 DEBUG [StoreOpener-fc3ebf4b2149ca5230c85524c0de4bba-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/f 2023-07-13 06:42:46,673 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:46,673 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a76e4d3d13ae5bb10f9d5a76e91300ce, NAME => 'hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce.', STARTKEY => '', ENDKEY => ''} 2023-07-13 06:42:46,674 INFO [StoreOpener-fc3ebf4b2149ca5230c85524c0de4bba-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fc3ebf4b2149ca5230c85524c0de4bba columnFamilyName f 2023-07-13 06:42:46,674 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:46,674 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 06:42:46,674 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:46,674 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:46,676 INFO [StoreOpener-a76e4d3d13ae5bb10f9d5a76e91300ce-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:46,679 DEBUG [StoreOpener-a76e4d3d13ae5bb10f9d5a76e91300ce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/info 2023-07-13 06:42:46,679 DEBUG [StoreOpener-a76e4d3d13ae5bb10f9d5a76e91300ce-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/info 2023-07-13 06:42:46,679 INFO [StoreOpener-a76e4d3d13ae5bb10f9d5a76e91300ce-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a76e4d3d13ae5bb10f9d5a76e91300ce columnFamilyName info 2023-07-13 06:42:46,680 INFO [StoreOpener-a76e4d3d13ae5bb10f9d5a76e91300ce-1] regionserver.HStore(310): Store=a76e4d3d13ae5bb10f9d5a76e91300ce/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 06:42:46,681 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:46,684 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:46,685 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/recovered.edits/0000000000000000005 2023-07-13 06:42:46,686 DEBUG [StoreOpener-fc3ebf4b2149ca5230c85524c0de4bba-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/f/9a13780ce1a446e6ae546f0a3c18c2dd 2023-07-13 06:42:46,687 INFO [StoreOpener-fc3ebf4b2149ca5230c85524c0de4bba-1] regionserver.HStore(310): Store=fc3ebf4b2149ca5230c85524c0de4bba/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 06:42:46,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:46,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:46,690 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/recovered.edits/0000000000000000005 2023-07-13 06:42:46,690 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing a76e4d3d13ae5bb10f9d5a76e91300ce 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-13 06:42:46,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:46,697 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-13 06:42:46,698 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened fc3ebf4b2149ca5230c85524c0de4bba; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73668151, jitterRate=0.09774099290370941}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-13 06:42:46,698 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for fc3ebf4b2149ca5230c85524c0de4bba: 2023-07-13 06:42:46,699 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba., pid=26, masterSystemTime=1689230566659 2023-07-13 06:42:46,701 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:46,702 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:46,702 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=fc3ebf4b2149ca5230c85524c0de4bba, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:46,702 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689230566702"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689230566702"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689230566702"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689230566702"}]},"ts":"1689230566702"} 2023-07-13 06:42:46,711 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/.tmp/info/20a4186057764dcf874044fb1691cf97 2023-07-13 06:42:46,712 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-07-13 06:42:46,712 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; OpenRegionProcedure fc3ebf4b2149ca5230c85524c0de4bba, server=jenkins-hbase20.apache.org,34261,1689230547862 in 199 msec 2023-07-13 06:42:46,714 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=t, region=fc3ebf4b2149ca5230c85524c0de4bba, ASSIGN in 474 msec 2023-07-13 06:42:46,720 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/.tmp/info/20a4186057764dcf874044fb1691cf97 as hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/info/20a4186057764dcf874044fb1691cf97 2023-07-13 06:42:46,726 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/info/20a4186057764dcf874044fb1691cf97, entries=2, sequenceid=5, filesize=4.8 K 2023-07-13 06:42:46,727 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for a76e4d3d13ae5bb10f9d5a76e91300ce in 37ms, sequenceid=5, compaction requested=false; wal=null 2023-07-13 06:42:46,728 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/recovered.edits/0000000000000000005 2023-07-13 06:42:46,732 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:46,734 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-13 06:42:46,735 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened a76e4d3d13ae5bb10f9d5a76e91300ce; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60528930, jitterRate=-0.09804865717887878}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-13 06:42:46,735 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for a76e4d3d13ae5bb10f9d5a76e91300ce: 2023-07-13 06:42:46,736 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce., pid=27, masterSystemTime=1689230566659 2023-07-13 06:42:46,738 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:46,738 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:46,738 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=a76e4d3d13ae5bb10f9d5a76e91300ce, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:46,739 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689230566738"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689230566738"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689230566738"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689230566738"}]},"ts":"1689230566738"} 2023-07-13 06:42:46,743 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=24 2023-07-13 06:42:46,743 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=24, state=SUCCESS; OpenRegionProcedure a76e4d3d13ae5bb10f9d5a76e91300ce, server=jenkins-hbase20.apache.org,34261,1689230547862 in 233 msec 2023-07-13 06:42:46,746 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=17 2023-07-13 06:42:46,746 INFO [PEWorker-2] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase20.apache.org,39919,1689230543088 after splitting done 2023-07-13 06:42:46,746 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=a76e4d3d13ae5bb10f9d5a76e91300ce, ASSIGN in 505 msec 2023-07-13 06:42:46,746 DEBUG [PEWorker-2] master.DeadServer(114): Removed jenkins-hbase20.apache.org,39919,1689230543088 from processing; numProcessing=0 2023-07-13 06:42:46,749 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=17, state=SUCCESS; ServerCrashProcedure jenkins-hbase20.apache.org,39919,1689230543088, splitWal=true, meta=true in 1.4330 sec 2023-07-13 06:42:46,865 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0539e0f9 to 127.0.0.1:58518 2023-07-13 06:42:46,865 DEBUG [Listener at localhost.localdomain/40319] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 06:42:46,865 INFO [Listener at localhost.localdomain/40319] client.TestScannerTimeout(124): END ************ test2772 2023-07-13 06:42:46,879 INFO [Listener at localhost.localdomain/40319] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=276 (was 295), OpenFileDescriptor=581 (was 571) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 204), ProcessCount=168 (was 168), AvailableMemoryMB=6133 (was 6141) 2023-07-13 06:42:46,879 INFO [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-13 06:42:46,880 INFO [Listener at localhost.localdomain/40319] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-13 06:42:46,880 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78e7a6a9 to 127.0.0.1:58518 2023-07-13 06:42:46,880 DEBUG [Listener at localhost.localdomain/40319] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 06:42:46,880 DEBUG [Listener at localhost.localdomain/40319] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-13 06:42:46,880 DEBUG [Listener at localhost.localdomain/40319] util.JVMClusterUtil(257): Found active master hash=656770925, stopped=false 2023-07-13 06:42:46,880 INFO [Listener at localhost.localdomain/40319] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,44209,1689230542090 2023-07-13 06:42:46,885 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-13 06:42:46,885 INFO [Listener at localhost.localdomain/40319] procedure2.ProcedureExecutor(629): Stopping 2023-07-13 06:42:46,885 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 06:42:46,885 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:34261-0x1008510fe3f0008, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-13 06:42:46,885 DEBUG [Listener at localhost.localdomain/40319] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x757253c7 to 127.0.0.1:58518 2023-07-13 06:42:46,885 DEBUG [Listener at localhost.localdomain/40319] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 06:42:46,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-13 06:42:46,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34261-0x1008510fe3f0008, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-13 06:42:46,886 INFO [Listener at localhost.localdomain/40319] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,34261,1689230547862' ***** 2023-07-13 06:42:46,886 INFO [Listener at localhost.localdomain/40319] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-13 06:42:46,886 INFO [RS:2;jenkins-hbase20:34261] regionserver.HeapMemoryManager(220): Stopping 2023-07-13 06:42:46,886 INFO [RS:2;jenkins-hbase20:34261] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-13 06:42:46,886 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-13 06:42:46,886 INFO [RS:2;jenkins-hbase20:34261] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-13 06:42:46,886 INFO [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(3305): Received CLOSE for a76e4d3d13ae5bb10f9d5a76e91300ce 2023-07-13 06:42:46,887 INFO [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(3305): Received CLOSE for fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:46,887 INFO [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:46,887 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing a76e4d3d13ae5bb10f9d5a76e91300ce, disabling compactions & flushes 2023-07-13 06:42:46,887 DEBUG [RS:2;jenkins-hbase20:34261] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a0e5441 to 127.0.0.1:58518 2023-07-13 06:42:46,887 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:46,887 DEBUG [RS:2;jenkins-hbase20:34261] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 06:42:46,887 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:46,887 INFO [RS:2;jenkins-hbase20:34261] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-13 06:42:46,887 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. after waiting 0 ms 2023-07-13 06:42:46,888 INFO [RS:2;jenkins-hbase20:34261] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-13 06:42:46,888 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:46,888 INFO [RS:2;jenkins-hbase20:34261] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-13 06:42:46,888 INFO [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-13 06:42:46,888 INFO [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-13 06:42:46,888 DEBUG [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(1478): Online Regions={a76e4d3d13ae5bb10f9d5a76e91300ce=hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce., 1588230740=hbase:meta,,1.1588230740, fc3ebf4b2149ca5230c85524c0de4bba=t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba.} 2023-07-13 06:42:46,888 DEBUG [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(1504): Waiting on 1588230740, a76e4d3d13ae5bb10f9d5a76e91300ce, fc3ebf4b2149ca5230c85524c0de4bba 2023-07-13 06:42:46,889 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-13 06:42:46,889 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-13 06:42:46,890 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-13 06:42:46,890 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-13 06:42:46,890 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-13 06:42:46,890 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-13 06:42:46,895 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/namespace/a76e4d3d13ae5bb10f9d5a76e91300ce/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=5 2023-07-13 06:42:46,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:46,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for a76e4d3d13ae5bb10f9d5a76e91300ce: 2023-07-13 06:42:46,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689230545019.a76e4d3d13ae5bb10f9d5a76e91300ce. 2023-07-13 06:42:46,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing fc3ebf4b2149ca5230c85524c0de4bba, disabling compactions & flushes 2023-07-13 06:42:46,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:46,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:46,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. after waiting 0 ms 2023-07-13 06:42:46,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:46,904 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/default/t/fc3ebf4b2149ca5230c85524c0de4bba/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-13 06:42:46,905 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:46,905 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for fc3ebf4b2149ca5230c85524c0de4bba: 2023-07-13 06:42:46,905 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed t,,1689230545830.fc3ebf4b2149ca5230c85524c0de4bba. 2023-07-13 06:42:46,920 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=22 (bloomFilter=false), to=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/.tmp/info/7e35d2c0e77b4985bf41dff332b7a209 2023-07-13 06:42:46,929 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/.tmp/info/7e35d2c0e77b4985bf41dff332b7a209 as hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/info/7e35d2c0e77b4985bf41dff332b7a209 2023-07-13 06:42:46,936 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/info/7e35d2c0e77b4985bf41dff332b7a209, entries=16, sequenceid=22, filesize=6.4 K 2023-07-13 06:42:46,938 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1796, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 48ms, sequenceid=22, compaction requested=false 2023-07-13 06:42:46,946 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/data/hbase/meta/1588230740/recovered.edits/25.seqid, newMaxSeqId=25, maxSeqId=15 2023-07-13 06:42:46,947 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-13 06:42:46,947 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-13 06:42:46,948 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-13 06:42:46,948 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-13 06:42:46,948 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-13 06:42:46,948 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-13 06:42:46,953 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-13 06:42:47,088 INFO [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,34261,1689230547862; all regions closed. 2023-07-13 06:42:47,094 DEBUG [RS:2;jenkins-hbase20:34261] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/oldWALs 2023-07-13 06:42:47,094 INFO [RS:2;jenkins-hbase20:34261] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C34261%2C1689230547862.meta:.meta(num 1689230565880) 2023-07-13 06:42:47,100 DEBUG [RS:2;jenkins-hbase20:34261] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/oldWALs 2023-07-13 06:42:47,100 INFO [RS:2;jenkins-hbase20:34261] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C34261%2C1689230547862:(num 1689230548084) 2023-07-13 06:42:47,100 DEBUG [RS:2;jenkins-hbase20:34261] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 06:42:47,100 INFO [RS:2;jenkins-hbase20:34261] regionserver.LeaseManager(133): Closed leases 2023-07-13 06:42:47,100 INFO [RS:2;jenkins-hbase20:34261] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-13 06:42:47,100 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-13 06:42:47,101 INFO [RS:2;jenkins-hbase20:34261] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:34261 2023-07-13 06:42:47,102 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 06:42:47,102 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:34261-0x1008510fe3f0008, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,34261,1689230547862 2023-07-13 06:42:47,103 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:34261-0x1008510fe3f0008, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 06:42:47,103 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,34261,1689230547862] 2023-07-13 06:42:47,103 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,34261,1689230547862; numProcessing=1 2023-07-13 06:42:47,104 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,34261,1689230547862 already deleted, retry=false 2023-07-13 06:42:47,104 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,34261,1689230547862 expired; onlineServers=0 2023-07-13 06:42:47,104 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,44209,1689230542090' ***** 2023-07-13 06:42:47,104 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-13 06:42:47,105 DEBUG [M:0;jenkins-hbase20:44209] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66188d56, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-13 06:42:47,105 INFO [M:0;jenkins-hbase20:44209] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,44209,1689230542090 2023-07-13 06:42:47,105 INFO [M:0;jenkins-hbase20:44209] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,44209,1689230542090; all regions closed. 2023-07-13 06:42:47,105 DEBUG [M:0;jenkins-hbase20:44209] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 06:42:47,105 DEBUG [M:0;jenkins-hbase20:44209] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-13 06:42:47,105 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-13 06:42:47,105 DEBUG [M:0;jenkins-hbase20:44209] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-13 06:42:47,105 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689230544196] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689230544196,5,FailOnTimeoutGroup] 2023-07-13 06:42:47,106 INFO [M:0;jenkins-hbase20:44209] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-13 06:42:47,106 INFO [M:0;jenkins-hbase20:44209] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-13 06:42:47,105 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689230544197] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689230544197,5,FailOnTimeoutGroup] 2023-07-13 06:42:47,106 INFO [M:0;jenkins-hbase20:44209] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-07-13 06:42:47,106 DEBUG [M:0;jenkins-hbase20:44209] master.HMaster(1512): Stopping service threads 2023-07-13 06:42:47,106 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-13 06:42:47,106 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 06:42:47,106 INFO [M:0;jenkins-hbase20:44209] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-13 06:42:47,106 ERROR [M:0;jenkins-hbase20:44209] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (910050368) connection to localhost.localdomain/127.0.0.1:46147 from jenkins,5,PEWorkerGroup] 2023-07-13 06:42:47,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-13 06:42:47,107 INFO [M:0;jenkins-hbase20:44209] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-13 06:42:47,107 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-13 06:42:47,107 DEBUG [M:0;jenkins-hbase20:44209] zookeeper.ZKUtil(398): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-13 06:42:47,107 WARN [M:0;jenkins-hbase20:44209] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-13 06:42:47,107 INFO [M:0;jenkins-hbase20:44209] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-13 06:42:47,108 INFO [M:0;jenkins-hbase20:44209] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-13 06:42:47,108 DEBUG [M:0;jenkins-hbase20:44209] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-13 06:42:47,108 INFO [M:0;jenkins-hbase20:44209] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-13 06:42:47,108 DEBUG [M:0;jenkins-hbase20:44209] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-13 06:42:47,108 DEBUG [M:0;jenkins-hbase20:44209] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-13 06:42:47,108 DEBUG [M:0;jenkins-hbase20:44209] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-13 06:42:47,108 INFO [M:0;jenkins-hbase20:44209] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=91.63 KB heapSize=110.36 KB 2023-07-13 06:42:47,124 INFO [M:0;jenkins-hbase20:44209] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=91.63 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e2bfbd6cec0f45b8b37eca1f632786a1 2023-07-13 06:42:47,131 DEBUG [M:0;jenkins-hbase20:44209] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e2bfbd6cec0f45b8b37eca1f632786a1 as hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e2bfbd6cec0f45b8b37eca1f632786a1 2023-07-13 06:42:47,139 INFO [M:0;jenkins-hbase20:44209] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46147/user/jenkins/test-data/f8b3a877-8dfb-3bea-93ee-d6052517e46c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e2bfbd6cec0f45b8b37eca1f632786a1, entries=27, sequenceid=212, filesize=8.2 K 2023-07-13 06:42:47,140 INFO [M:0;jenkins-hbase20:44209] regionserver.HRegion(2948): Finished flush of dataSize ~91.63 KB/93824, heapSize ~110.34 KB/112992, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 32ms, sequenceid=212, compaction requested=false 2023-07-13 06:42:47,141 INFO [M:0;jenkins-hbase20:44209] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-13 06:42:47,141 DEBUG [M:0;jenkins-hbase20:44209] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-13 06:42:47,145 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-13 06:42:47,145 INFO [M:0;jenkins-hbase20:44209] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-13 06:42:47,146 INFO [M:0;jenkins-hbase20:44209] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:44209 2023-07-13 06:42:47,147 DEBUG [M:0;jenkins-hbase20:44209] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,44209,1689230542090 already deleted, retry=false 2023-07-13 06:42:47,370 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-13 06:42:47,370 INFO [M:0;jenkins-hbase20:44209] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,44209,1689230542090; zookeeper connection closed. 2023-07-13 06:42:47,370 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): master:44209-0x1008510fe3f0000, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-13 06:42:47,471 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:34261-0x1008510fe3f0008, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-13 06:42:47,471 INFO [RS:2;jenkins-hbase20:34261] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,34261,1689230547862; zookeeper connection closed. 2023-07-13 06:42:47,471 DEBUG [Listener at localhost.localdomain/40319-EventThread] zookeeper.ZKWatcher(600): regionserver:34261-0x1008510fe3f0008, quorum=127.0.0.1:58518, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-13 06:42:47,471 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@75ba4ac9] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@75ba4ac9 2023-07-13 06:42:47,472 INFO [Listener at localhost.localdomain/40319] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-13 06:42:47,472 WARN [Listener at localhost.localdomain/40319] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-13 06:42:47,479 INFO [Listener at localhost.localdomain/40319] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-13 06:42:47,589 WARN [BP-578202921-148.251.75.209-1689230539315 heartbeating to localhost.localdomain/127.0.0.1:46147] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-13 06:42:47,589 WARN [BP-578202921-148.251.75.209-1689230539315 heartbeating to localhost.localdomain/127.0.0.1:46147] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-578202921-148.251.75.209-1689230539315 (Datanode Uuid cdebc699-f457-4676-abe1-0db4364b2ae5) service to localhost.localdomain/127.0.0.1:46147 2023-07-13 06:42:47,591 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/cluster_8d5ebd29-c8d7-9901-4c53-f6c110004b44/dfs/data/data3/current/BP-578202921-148.251.75.209-1689230539315] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-13 06:42:47,592 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/cluster_8d5ebd29-c8d7-9901-4c53-f6c110004b44/dfs/data/data4/current/BP-578202921-148.251.75.209-1689230539315] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-13 06:42:47,594 WARN [Listener at localhost.localdomain/40319] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-13 06:42:47,600 INFO [Listener at localhost.localdomain/40319] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-13 06:42:47,709 WARN [BP-578202921-148.251.75.209-1689230539315 heartbeating to localhost.localdomain/127.0.0.1:46147] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-13 06:42:47,709 WARN [BP-578202921-148.251.75.209-1689230539315 heartbeating to localhost.localdomain/127.0.0.1:46147] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-578202921-148.251.75.209-1689230539315 (Datanode Uuid 305f01f8-519a-4fc6-a5d1-b59b788bbb0f) service to localhost.localdomain/127.0.0.1:46147 2023-07-13 06:42:47,710 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/cluster_8d5ebd29-c8d7-9901-4c53-f6c110004b44/dfs/data/data1/current/BP-578202921-148.251.75.209-1689230539315] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-13 06:42:47,711 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b02ce939-ba8c-5d5b-94ba-0f6861ee9370/cluster_8d5ebd29-c8d7-9901-4c53-f6c110004b44/dfs/data/data2/current/BP-578202921-148.251.75.209-1689230539315] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-13 06:42:47,743 INFO [Listener at localhost.localdomain/40319] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-07-13 06:42:47,855 INFO [Listener at localhost.localdomain/40319] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-13 06:42:47,895 INFO [Listener at localhost.localdomain/40319] hbase.HBaseTestingUtility(1293): Minicluster is down