2023-07-12 18:43:13,121 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-12 18:43:13,322 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768 2023-07-12 18:43:13,334 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-12 18:43:13,335 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/cluster_55ffc31b-bad0-f043-ae3b-8971827a1352, deleteOnExit=true 2023-07-12 18:43:13,335 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-12 18:43:13,335 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/test.cache.data in system properties and HBase conf 2023-07-12 18:43:13,336 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 18:43:13,336 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/hadoop.log.dir in system properties and HBase conf 2023-07-12 18:43:13,337 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 18:43:13,337 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 18:43:13,337 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 18:43:13,432 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-12 18:43:13,791 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 18:43:13,794 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 18:43:13,795 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 18:43:13,795 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 18:43:13,795 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 18:43:13,796 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 18:43:13,796 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 18:43:13,796 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 18:43:13,796 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 18:43:13,797 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 18:43:13,797 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/nfs.dump.dir in system properties and HBase conf 2023-07-12 18:43:13,797 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/java.io.tmpdir in system properties and HBase conf 2023-07-12 18:43:13,798 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 18:43:13,798 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 18:43:13,798 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 18:43:14,201 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 18:43:14,205 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 18:43:14,423 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-12 18:43:14,565 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-12 18:43:14,578 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 18:43:14,615 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 18:43:14,643 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/java.io.tmpdir/Jetty_localhost_localdomain_35081_hdfs____5sgnkx/webapp 2023-07-12 18:43:14,783 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:35081 2023-07-12 18:43:14,791 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 18:43:14,792 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 18:43:15,144 WARN [Listener at localhost.localdomain/33493] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 18:43:15,227 WARN [Listener at localhost.localdomain/33493] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 18:43:15,242 WARN [Listener at localhost.localdomain/33493] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 18:43:15,249 INFO [Listener at localhost.localdomain/33493] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 18:43:15,253 INFO [Listener at localhost.localdomain/33493] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/java.io.tmpdir/Jetty_localhost_36263_datanode____.3q9a7h/webapp 2023-07-12 18:43:15,330 INFO [Listener at localhost.localdomain/33493] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36263 2023-07-12 18:43:15,628 WARN [Listener at localhost.localdomain/43667] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 18:43:15,639 WARN [Listener at localhost.localdomain/43667] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 18:43:15,642 WARN [Listener at localhost.localdomain/43667] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 18:43:15,643 INFO [Listener at localhost.localdomain/43667] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 18:43:15,651 INFO [Listener at localhost.localdomain/43667] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/java.io.tmpdir/Jetty_localhost_36553_datanode____8eaqlz/webapp 2023-07-12 18:43:15,740 INFO [Listener at localhost.localdomain/43667] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36553 2023-07-12 18:43:15,750 WARN [Listener at localhost.localdomain/46185] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 18:43:16,007 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x39bd4e26b2dcda9e: Processing first storage report for DS-3d7629ec-da28-4d9a-b55e-6763e4437bb1 from datanode bc0640a0-7a83-4df7-8de8-a7ab5411d425 2023-07-12 18:43:16,008 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x39bd4e26b2dcda9e: from storage DS-3d7629ec-da28-4d9a-b55e-6763e4437bb1 node DatanodeRegistration(127.0.0.1:45039, datanodeUuid=bc0640a0-7a83-4df7-8de8-a7ab5411d425, infoPort=45141, infoSecurePort=0, ipcPort=43667, storageInfo=lv=-57;cid=testClusterID;nsid=2112607822;c=1689187394261), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-12 18:43:16,008 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x24c9ce73b542cf43: Processing first storage report for DS-4ad92ce8-6d56-445f-9b36-b8468e44fc58 from datanode 019ccb5f-98d9-478b-9e9b-908bbab7e733 2023-07-12 18:43:16,008 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x24c9ce73b542cf43: from storage DS-4ad92ce8-6d56-445f-9b36-b8468e44fc58 node DatanodeRegistration(127.0.0.1:42053, datanodeUuid=019ccb5f-98d9-478b-9e9b-908bbab7e733, infoPort=32821, infoSecurePort=0, ipcPort=46185, storageInfo=lv=-57;cid=testClusterID;nsid=2112607822;c=1689187394261), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 18:43:16,008 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x39bd4e26b2dcda9e: Processing first storage report for DS-e05089c0-1327-49eb-92c7-8f07b881e2eb from datanode bc0640a0-7a83-4df7-8de8-a7ab5411d425 2023-07-12 18:43:16,008 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x39bd4e26b2dcda9e: from storage DS-e05089c0-1327-49eb-92c7-8f07b881e2eb node DatanodeRegistration(127.0.0.1:45039, datanodeUuid=bc0640a0-7a83-4df7-8de8-a7ab5411d425, infoPort=45141, infoSecurePort=0, ipcPort=43667, storageInfo=lv=-57;cid=testClusterID;nsid=2112607822;c=1689187394261), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 18:43:16,009 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x24c9ce73b542cf43: Processing first storage report for DS-5bb124be-c487-4cb9-b83d-90ae048e8f2e from datanode 019ccb5f-98d9-478b-9e9b-908bbab7e733 2023-07-12 18:43:16,009 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x24c9ce73b542cf43: from storage DS-5bb124be-c487-4cb9-b83d-90ae048e8f2e node DatanodeRegistration(127.0.0.1:42053, datanodeUuid=019ccb5f-98d9-478b-9e9b-908bbab7e733, infoPort=32821, infoSecurePort=0, ipcPort=46185, storageInfo=lv=-57;cid=testClusterID;nsid=2112607822;c=1689187394261), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 18:43:16,089 DEBUG [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768 2023-07-12 18:43:16,142 INFO [Listener at localhost.localdomain/46185] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/cluster_55ffc31b-bad0-f043-ae3b-8971827a1352/zookeeper_0, clientPort=63743, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/cluster_55ffc31b-bad0-f043-ae3b-8971827a1352/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/cluster_55ffc31b-bad0-f043-ae3b-8971827a1352/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 18:43:16,154 INFO [Listener at localhost.localdomain/46185] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=63743 2023-07-12 18:43:16,165 INFO [Listener at localhost.localdomain/46185] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 18:43:16,168 INFO [Listener at localhost.localdomain/46185] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 18:43:16,784 INFO [Listener at localhost.localdomain/46185] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d with version=8 2023-07-12 18:43:16,784 INFO [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/hbase-staging 2023-07-12 18:43:17,027 INFO [Listener at localhost.localdomain/46185] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-12 18:43:17,382 INFO [Listener at localhost.localdomain/46185] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 18:43:17,407 INFO [Listener at localhost.localdomain/46185] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 18:43:17,407 INFO [Listener at localhost.localdomain/46185] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 18:43:17,407 INFO [Listener at localhost.localdomain/46185] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 18:43:17,407 INFO [Listener at localhost.localdomain/46185] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 18:43:17,408 INFO [Listener at localhost.localdomain/46185] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 18:43:17,520 INFO [Listener at localhost.localdomain/46185] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 18:43:17,577 DEBUG [Listener at localhost.localdomain/46185] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-12 18:43:17,660 INFO [Listener at localhost.localdomain/46185] ipc.NettyRpcServer(120): Bind to /148.251.75.209:33301 2023-07-12 18:43:17,669 INFO [Listener at localhost.localdomain/46185] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 18:43:17,671 INFO [Listener at localhost.localdomain/46185] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 18:43:17,688 INFO [Listener at localhost.localdomain/46185] zookeeper.RecoverableZooKeeper(93): Process identifier=master:33301 connecting to ZooKeeper ensemble=127.0.0.1:63743 2023-07-12 18:43:17,720 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:333010x0, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 18:43:17,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:33301-0x100827ea6a00000 connected 2023-07-12 18:43:17,745 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ZKUtil(164): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 18:43:17,746 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ZKUtil(164): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 18:43:17,750 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ZKUtil(164): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 18:43:17,758 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33301 2023-07-12 18:43:17,759 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33301 2023-07-12 18:43:17,759 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33301 2023-07-12 18:43:17,759 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33301 2023-07-12 18:43:17,760 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33301 2023-07-12 18:43:17,764 INFO [Listener at localhost.localdomain/46185] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d, hbase.cluster.distributed=false 2023-07-12 18:43:17,822 INFO [Listener at localhost.localdomain/46185] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 18:43:17,823 INFO [Listener at localhost.localdomain/46185] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 18:43:17,823 INFO [Listener at localhost.localdomain/46185] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 18:43:17,823 INFO [Listener at localhost.localdomain/46185] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 18:43:17,823 INFO [Listener at localhost.localdomain/46185] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 18:43:17,823 INFO [Listener at localhost.localdomain/46185] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 18:43:17,827 INFO [Listener at localhost.localdomain/46185] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 18:43:17,830 INFO [Listener at localhost.localdomain/46185] ipc.NettyRpcServer(120): Bind to /148.251.75.209:39999 2023-07-12 18:43:17,832 INFO [Listener at localhost.localdomain/46185] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 18:43:17,838 DEBUG [Listener at localhost.localdomain/46185] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 18:43:17,839 INFO [Listener at localhost.localdomain/46185] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 18:43:17,842 INFO [Listener at localhost.localdomain/46185] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 18:43:17,843 INFO [Listener at localhost.localdomain/46185] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39999 connecting to ZooKeeper ensemble=127.0.0.1:63743 2023-07-12 18:43:17,847 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:399990x0, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 18:43:17,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:39999-0x100827ea6a00001 connected 2023-07-12 18:43:17,849 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ZKUtil(164): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 18:43:17,850 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ZKUtil(164): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 18:43:17,851 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ZKUtil(164): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 18:43:17,851 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39999 2023-07-12 18:43:17,851 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39999 2023-07-12 18:43:17,853 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39999 2023-07-12 18:43:17,854 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39999 2023-07-12 18:43:17,854 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39999 2023-07-12 18:43:17,866 INFO [Listener at localhost.localdomain/46185] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 18:43:17,867 INFO [Listener at localhost.localdomain/46185] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 18:43:17,867 INFO [Listener at localhost.localdomain/46185] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 18:43:17,867 INFO [Listener at localhost.localdomain/46185] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 18:43:17,867 INFO [Listener at localhost.localdomain/46185] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 18:43:17,868 INFO [Listener at localhost.localdomain/46185] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 18:43:17,868 INFO [Listener at localhost.localdomain/46185] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 18:43:17,871 INFO [Listener at localhost.localdomain/46185] ipc.NettyRpcServer(120): Bind to /148.251.75.209:46475 2023-07-12 18:43:17,872 INFO [Listener at localhost.localdomain/46185] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 18:43:17,873 DEBUG [Listener at localhost.localdomain/46185] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 18:43:17,874 INFO [Listener at localhost.localdomain/46185] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 18:43:17,876 INFO [Listener at localhost.localdomain/46185] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 18:43:17,879 INFO [Listener at localhost.localdomain/46185] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46475 connecting to ZooKeeper ensemble=127.0.0.1:63743 2023-07-12 18:43:17,884 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:464750x0, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 18:43:17,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:46475-0x100827ea6a00002 connected 2023-07-12 18:43:17,885 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ZKUtil(164): regionserver:46475-0x100827ea6a00002, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 18:43:17,886 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ZKUtil(164): regionserver:46475-0x100827ea6a00002, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 18:43:17,887 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ZKUtil(164): regionserver:46475-0x100827ea6a00002, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 18:43:17,887 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46475 2023-07-12 18:43:17,887 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46475 2023-07-12 18:43:17,888 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46475 2023-07-12 18:43:17,888 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46475 2023-07-12 18:43:17,888 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46475 2023-07-12 18:43:17,889 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,33301,1689187396898 2023-07-12 18:43:17,901 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 18:43:17,902 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,33301,1689187396898 2023-07-12 18:43:17,921 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 18:43:17,921 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 18:43:17,921 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:46475-0x100827ea6a00002, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 18:43:17,921 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 18:43:17,923 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 18:43:17,924 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,33301,1689187396898 from backup master directory 2023-07-12 18:43:17,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 18:43:17,927 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,33301,1689187396898 2023-07-12 18:43:17,927 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 18:43:17,927 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 18:43:17,927 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,33301,1689187396898 2023-07-12 18:43:17,930 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-12 18:43:17,931 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-12 18:43:18,015 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/hbase.id with ID: abb7fc94-1c96-4221-9e93-46bbae4759c5 2023-07-12 18:43:18,063 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 18:43:18,081 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 18:43:18,123 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0007cd6a to 127.0.0.1:63743 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 18:43:18,152 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a423a9a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 18:43:18,170 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 18:43:18,172 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 18:43:18,186 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-12 18:43:18,186 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-12 18:43:18,187 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 18:43:18,191 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 18:43:18,192 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 18:43:18,219 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/MasterData/data/master/store-tmp 2023-07-12 18:43:18,248 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 18:43:18,248 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 18:43:18,248 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 18:43:18,248 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 18:43:18,249 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 18:43:18,249 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 18:43:18,249 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 18:43:18,249 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 18:43:18,250 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/MasterData/WALs/jenkins-hbase20.apache.org,33301,1689187396898 2023-07-12 18:43:18,270 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C33301%2C1689187396898, suffix=, logDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/MasterData/WALs/jenkins-hbase20.apache.org,33301,1689187396898, archiveDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/MasterData/oldWALs, maxLogs=10 2023-07-12 18:43:18,317 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42053,DS-4ad92ce8-6d56-445f-9b36-b8468e44fc58,DISK] 2023-07-12 18:43:18,317 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45039,DS-3d7629ec-da28-4d9a-b55e-6763e4437bb1,DISK] 2023-07-12 18:43:18,324 DEBUG [RS-EventLoopGroup-4-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 18:43:18,380 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/MasterData/WALs/jenkins-hbase20.apache.org,33301,1689187396898/jenkins-hbase20.apache.org%2C33301%2C1689187396898.1689187398278 2023-07-12 18:43:18,380 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42053,DS-4ad92ce8-6d56-445f-9b36-b8468e44fc58,DISK], DatanodeInfoWithStorage[127.0.0.1:45039,DS-3d7629ec-da28-4d9a-b55e-6763e4437bb1,DISK]] 2023-07-12 18:43:18,381 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 18:43:18,381 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 18:43:18,384 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 18:43:18,385 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 18:43:18,435 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 18:43:18,443 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 18:43:18,463 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 18:43:18,474 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 18:43:18,480 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 18:43:18,481 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 18:43:18,496 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 18:43:18,501 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 18:43:18,502 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69887183, jitterRate=0.041400179266929626}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 18:43:18,502 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 18:43:18,504 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 18:43:18,520 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-12 18:43:18,520 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-12 18:43:18,522 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 18:43:18,524 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-12 18:43:18,550 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 25 msec 2023-07-12 18:43:18,550 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 18:43:18,571 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 18:43:18,575 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 18:43:18,602 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 18:43:18,606 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 18:43:18,608 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 18:43:18,613 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 18:43:18,617 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 18:43:18,620 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 18:43:18,621 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 18:43:18,622 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 18:43:18,633 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 18:43:18,636 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 18:43:18,636 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 18:43:18,636 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:46475-0x100827ea6a00002, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 18:43:18,637 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 18:43:18,637 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,33301,1689187396898, sessionid=0x100827ea6a00000, setting cluster-up flag (Was=false) 2023-07-12 18:43:18,652 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 18:43:18,656 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 18:43:18,658 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,33301,1689187396898 2023-07-12 18:43:18,662 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 18:43:18,666 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 18:43:18,667 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,33301,1689187396898 2023-07-12 18:43:18,669 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/.hbase-snapshot/.tmp 2023-07-12 18:43:18,691 INFO [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(951): ClusterId : abb7fc94-1c96-4221-9e93-46bbae4759c5 2023-07-12 18:43:18,691 INFO [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(951): ClusterId : abb7fc94-1c96-4221-9e93-46bbae4759c5 2023-07-12 18:43:18,697 DEBUG [RS:0;jenkins-hbase20:39999] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 18:43:18,697 DEBUG [RS:1;jenkins-hbase20:46475] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 18:43:18,702 DEBUG [RS:1;jenkins-hbase20:46475] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 18:43:18,702 DEBUG [RS:0;jenkins-hbase20:39999] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 18:43:18,702 DEBUG [RS:1;jenkins-hbase20:46475] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 18:43:18,702 DEBUG [RS:0;jenkins-hbase20:39999] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 18:43:18,704 DEBUG [RS:1;jenkins-hbase20:46475] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 18:43:18,704 DEBUG [RS:0;jenkins-hbase20:39999] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 18:43:18,706 DEBUG [RS:0;jenkins-hbase20:39999] zookeeper.ReadOnlyZKClient(139): Connect 0x7310b462 to 127.0.0.1:63743 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 18:43:18,706 DEBUG [RS:1;jenkins-hbase20:46475] zookeeper.ReadOnlyZKClient(139): Connect 0x19b6a5aa to 127.0.0.1:63743 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 18:43:18,718 DEBUG [RS:1;jenkins-hbase20:46475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@177158ec, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 18:43:18,718 DEBUG [RS:0;jenkins-hbase20:39999] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7dd74470, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 18:43:18,719 DEBUG [RS:1;jenkins-hbase20:46475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@134eceda, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 18:43:18,719 DEBUG [RS:0;jenkins-hbase20:39999] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79e8df12, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 18:43:18,741 DEBUG [RS:0;jenkins-hbase20:39999] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:39999 2023-07-12 18:43:18,743 DEBUG [RS:1;jenkins-hbase20:46475] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase20:46475 2023-07-12 18:43:18,745 INFO [RS:0;jenkins-hbase20:39999] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 18:43:18,745 INFO [RS:0;jenkins-hbase20:39999] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 18:43:18,745 INFO [RS:1;jenkins-hbase20:46475] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 18:43:18,746 INFO [RS:1;jenkins-hbase20:46475] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 18:43:18,746 DEBUG [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 18:43:18,745 DEBUG [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 18:43:18,748 INFO [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,33301,1689187396898 with isa=jenkins-hbase20.apache.org/148.251.75.209:39999, startcode=1689187397821 2023-07-12 18:43:18,748 INFO [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,33301,1689187396898 with isa=jenkins-hbase20.apache.org/148.251.75.209:46475, startcode=1689187397866 2023-07-12 18:43:18,764 DEBUG [RS:0;jenkins-hbase20:39999] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 18:43:18,764 DEBUG [RS:1;jenkins-hbase20:46475] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 18:43:18,779 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 18:43:18,794 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 18:43:18,795 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 18:43:18,795 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 18:43:18,795 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-12 18:43:18,795 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-07-12 18:43:18,796 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:18,796 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 18:43:18,796 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:18,798 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689187428798 2023-07-12 18:43:18,800 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 18:43:18,806 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 18:43:18,807 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 18:43:18,813 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 18:43:18,813 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 18:43:18,820 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58825, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 18:43:18,820 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 18:43:18,820 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60147, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 18:43:18,821 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 18:43:18,822 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 18:43:18,822 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 18:43:18,822 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:18,824 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 18:43:18,826 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 18:43:18,826 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 18:43:18,828 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 18:43:18,828 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 18:43:18,830 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33301] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:579) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 18:43:18,830 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689187398829,5,FailOnTimeoutGroup] 2023-07-12 18:43:18,831 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689187398831,5,FailOnTimeoutGroup] 2023-07-12 18:43:18,831 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:18,831 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 18:43:18,832 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:18,832 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:18,847 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33301] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:18,857 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 18:43:18,858 DEBUG [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d 2023-07-12 18:43:18,858 DEBUG [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:33493 2023-07-12 18:43:18,858 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 18:43:18,858 DEBUG [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 18:43:18,859 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d 2023-07-12 18:43:18,867 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 18:43:18,867 DEBUG [RS:0;jenkins-hbase20:39999] zookeeper.ZKUtil(162): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:18,868 WARN [RS:0;jenkins-hbase20:39999] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 18:43:18,868 INFO [RS:0;jenkins-hbase20:39999] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 18:43:18,869 DEBUG [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:18,871 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,39999,1689187397821] 2023-07-12 18:43:18,875 DEBUG [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(2830): Master is not running yet 2023-07-12 18:43:18,876 WARN [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(1030): reportForDuty failed; sleeping 100 ms and then retrying. 2023-07-12 18:43:18,887 DEBUG [RS:0;jenkins-hbase20:39999] zookeeper.ZKUtil(162): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:18,887 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 18:43:18,890 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 18:43:18,894 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/info 2023-07-12 18:43:18,895 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 18:43:18,897 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 18:43:18,897 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 18:43:18,900 DEBUG [RS:0;jenkins-hbase20:39999] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 18:43:18,900 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/rep_barrier 2023-07-12 18:43:18,901 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 18:43:18,902 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 18:43:18,902 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 18:43:18,904 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/table 2023-07-12 18:43:18,905 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 18:43:18,905 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 18:43:18,907 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740 2023-07-12 18:43:18,908 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740 2023-07-12 18:43:18,909 INFO [RS:0;jenkins-hbase20:39999] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 18:43:18,912 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 18:43:18,914 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 18:43:18,918 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 18:43:18,919 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63200116, jitterRate=-0.05824488401412964}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 18:43:18,919 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 18:43:18,919 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 18:43:18,919 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 18:43:18,919 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 18:43:18,919 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 18:43:18,919 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 18:43:18,920 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 18:43:18,921 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 18:43:18,927 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 18:43:18,927 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 18:43:18,934 INFO [RS:0;jenkins-hbase20:39999] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 18:43:18,936 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 18:43:18,939 INFO [RS:0;jenkins-hbase20:39999] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 18:43:18,939 INFO [RS:0;jenkins-hbase20:39999] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:18,940 INFO [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 18:43:18,948 INFO [RS:0;jenkins-hbase20:39999] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:18,948 DEBUG [RS:0;jenkins-hbase20:39999] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:18,948 DEBUG [RS:0;jenkins-hbase20:39999] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:18,948 DEBUG [RS:0;jenkins-hbase20:39999] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:18,948 DEBUG [RS:0;jenkins-hbase20:39999] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:18,949 DEBUG [RS:0;jenkins-hbase20:39999] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:18,949 DEBUG [RS:0;jenkins-hbase20:39999] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 18:43:18,949 DEBUG [RS:0;jenkins-hbase20:39999] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:18,949 DEBUG [RS:0;jenkins-hbase20:39999] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:18,949 DEBUG [RS:0;jenkins-hbase20:39999] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:18,950 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 18:43:18,950 DEBUG [RS:0;jenkins-hbase20:39999] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:18,952 INFO [RS:0;jenkins-hbase20:39999] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:18,952 INFO [RS:0;jenkins-hbase20:39999] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:18,952 INFO [RS:0;jenkins-hbase20:39999] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:18,953 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 18:43:18,965 INFO [RS:0;jenkins-hbase20:39999] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 18:43:18,967 INFO [RS:0;jenkins-hbase20:39999] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,39999,1689187397821-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:18,977 INFO [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,33301,1689187396898 with isa=jenkins-hbase20.apache.org/148.251.75.209:46475, startcode=1689187397866 2023-07-12 18:43:18,978 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33301] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,46475,1689187397866 2023-07-12 18:43:18,980 DEBUG [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d 2023-07-12 18:43:18,980 DEBUG [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:33493 2023-07-12 18:43:18,980 DEBUG [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 18:43:18,982 INFO [RS:0;jenkins-hbase20:39999] regionserver.Replication(203): jenkins-hbase20.apache.org,39999,1689187397821 started 2023-07-12 18:43:18,982 INFO [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,39999,1689187397821, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:39999, sessionid=0x100827ea6a00001 2023-07-12 18:43:18,982 DEBUG [RS:0;jenkins-hbase20:39999] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 18:43:18,982 DEBUG [RS:0;jenkins-hbase20:39999] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:18,982 DEBUG [RS:0;jenkins-hbase20:39999] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,39999,1689187397821' 2023-07-12 18:43:18,982 DEBUG [RS:0;jenkins-hbase20:39999] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 18:43:18,982 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 18:43:18,983 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 18:43:18,983 DEBUG [RS:1;jenkins-hbase20:46475] zookeeper.ZKUtil(162): regionserver:46475-0x100827ea6a00002, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,46475,1689187397866 2023-07-12 18:43:18,983 DEBUG [RS:0;jenkins-hbase20:39999] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 18:43:18,983 WARN [RS:1;jenkins-hbase20:46475] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 18:43:18,983 INFO [RS:1;jenkins-hbase20:46475] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 18:43:18,983 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,46475,1689187397866] 2023-07-12 18:43:18,983 DEBUG [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866 2023-07-12 18:43:18,984 DEBUG [RS:0;jenkins-hbase20:39999] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 18:43:18,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:18,984 DEBUG [RS:0;jenkins-hbase20:39999] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 18:43:18,984 DEBUG [RS:0;jenkins-hbase20:39999] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:18,985 DEBUG [RS:0;jenkins-hbase20:39999] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,39999,1689187397821' 2023-07-12 18:43:18,985 DEBUG [RS:0;jenkins-hbase20:39999] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 18:43:18,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,46475,1689187397866 2023-07-12 18:43:18,986 DEBUG [RS:0;jenkins-hbase20:39999] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 18:43:18,987 DEBUG [RS:0;jenkins-hbase20:39999] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 18:43:18,987 INFO [RS:0;jenkins-hbase20:39999] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 18:43:18,987 INFO [RS:0;jenkins-hbase20:39999] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 18:43:18,989 DEBUG [RS:1;jenkins-hbase20:46475] zookeeper.ZKUtil(162): regionserver:46475-0x100827ea6a00002, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:18,989 DEBUG [RS:1;jenkins-hbase20:46475] zookeeper.ZKUtil(162): regionserver:46475-0x100827ea6a00002, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,46475,1689187397866 2023-07-12 18:43:18,990 DEBUG [RS:1;jenkins-hbase20:46475] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 18:43:18,991 INFO [RS:1;jenkins-hbase20:46475] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 18:43:18,995 INFO [RS:1;jenkins-hbase20:46475] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 18:43:18,996 INFO [RS:1;jenkins-hbase20:46475] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 18:43:18,997 INFO [RS:1;jenkins-hbase20:46475] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:18,997 INFO [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 18:43:18,999 INFO [RS:1;jenkins-hbase20:46475] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:18,999 DEBUG [RS:1;jenkins-hbase20:46475] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:19,000 DEBUG [RS:1;jenkins-hbase20:46475] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:19,000 DEBUG [RS:1;jenkins-hbase20:46475] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:19,000 DEBUG [RS:1;jenkins-hbase20:46475] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:19,000 DEBUG [RS:1;jenkins-hbase20:46475] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:19,001 DEBUG [RS:1;jenkins-hbase20:46475] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 18:43:19,001 DEBUG [RS:1;jenkins-hbase20:46475] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:19,001 DEBUG [RS:1;jenkins-hbase20:46475] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:19,001 DEBUG [RS:1;jenkins-hbase20:46475] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:19,001 DEBUG [RS:1;jenkins-hbase20:46475] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:19,002 INFO [RS:1;jenkins-hbase20:46475] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:19,002 INFO [RS:1;jenkins-hbase20:46475] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:19,002 INFO [RS:1;jenkins-hbase20:46475] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:19,012 INFO [RS:1;jenkins-hbase20:46475] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 18:43:19,013 INFO [RS:1;jenkins-hbase20:46475] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46475,1689187397866-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:19,022 INFO [RS:1;jenkins-hbase20:46475] regionserver.Replication(203): jenkins-hbase20.apache.org,46475,1689187397866 started 2023-07-12 18:43:19,022 INFO [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,46475,1689187397866, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:46475, sessionid=0x100827ea6a00002 2023-07-12 18:43:19,022 DEBUG [RS:1;jenkins-hbase20:46475] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 18:43:19,022 DEBUG [RS:1;jenkins-hbase20:46475] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,46475,1689187397866 2023-07-12 18:43:19,022 DEBUG [RS:1;jenkins-hbase20:46475] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,46475,1689187397866' 2023-07-12 18:43:19,023 DEBUG [RS:1;jenkins-hbase20:46475] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 18:43:19,024 DEBUG [RS:1;jenkins-hbase20:46475] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 18:43:19,024 DEBUG [RS:1;jenkins-hbase20:46475] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 18:43:19,024 DEBUG [RS:1;jenkins-hbase20:46475] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 18:43:19,024 DEBUG [RS:1;jenkins-hbase20:46475] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,46475,1689187397866 2023-07-12 18:43:19,025 DEBUG [RS:1;jenkins-hbase20:46475] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,46475,1689187397866' 2023-07-12 18:43:19,025 DEBUG [RS:1;jenkins-hbase20:46475] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 18:43:19,025 DEBUG [RS:1;jenkins-hbase20:46475] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 18:43:19,026 DEBUG [RS:1;jenkins-hbase20:46475] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 18:43:19,026 INFO [RS:1;jenkins-hbase20:46475] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 18:43:19,026 INFO [RS:1;jenkins-hbase20:46475] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 18:43:19,098 INFO [RS:0;jenkins-hbase20:39999] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C39999%2C1689187397821, suffix=, logDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821, archiveDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/oldWALs, maxLogs=32 2023-07-12 18:43:19,105 DEBUG [jenkins-hbase20:33301] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-12 18:43:19,109 DEBUG [jenkins-hbase20:33301] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 18:43:19,116 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42053,DS-4ad92ce8-6d56-445f-9b36-b8468e44fc58,DISK] 2023-07-12 18:43:19,116 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45039,DS-3d7629ec-da28-4d9a-b55e-6763e4437bb1,DISK] 2023-07-12 18:43:19,118 DEBUG [jenkins-hbase20:33301] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 18:43:19,118 DEBUG [jenkins-hbase20:33301] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 18:43:19,118 DEBUG [jenkins-hbase20:33301] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 18:43:19,120 INFO [RS:0;jenkins-hbase20:39999] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821/jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100 2023-07-12 18:43:19,120 DEBUG [RS:0;jenkins-hbase20:39999] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42053,DS-4ad92ce8-6d56-445f-9b36-b8468e44fc58,DISK], DatanodeInfoWithStorage[127.0.0.1:45039,DS-3d7629ec-da28-4d9a-b55e-6763e4437bb1,DISK]] 2023-07-12 18:43:19,121 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,39999,1689187397821, state=OPENING 2023-07-12 18:43:19,127 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 18:43:19,128 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 18:43:19,129 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 18:43:19,130 INFO [RS:1;jenkins-hbase20:46475] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C46475%2C1689187397866, suffix=, logDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866, archiveDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/oldWALs, maxLogs=32 2023-07-12 18:43:19,134 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,39999,1689187397821}] 2023-07-12 18:43:19,154 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42053,DS-4ad92ce8-6d56-445f-9b36-b8468e44fc58,DISK] 2023-07-12 18:43:19,154 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45039,DS-3d7629ec-da28-4d9a-b55e-6763e4437bb1,DISK] 2023-07-12 18:43:19,157 INFO [RS:1;jenkins-hbase20:46475] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866/jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133 2023-07-12 18:43:19,158 DEBUG [RS:1;jenkins-hbase20:46475] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42053,DS-4ad92ce8-6d56-445f-9b36-b8468e44fc58,DISK], DatanodeInfoWithStorage[127.0.0.1:45039,DS-3d7629ec-da28-4d9a-b55e-6763e4437bb1,DISK]] 2023-07-12 18:43:19,319 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:19,321 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 18:43:19,324 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34176, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 18:43:19,336 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 18:43:19,337 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 18:43:19,341 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821, archiveDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/oldWALs, maxLogs=32 2023-07-12 18:43:19,357 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45039,DS-3d7629ec-da28-4d9a-b55e-6763e4437bb1,DISK] 2023-07-12 18:43:19,357 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42053,DS-4ad92ce8-6d56-445f-9b36-b8468e44fc58,DISK] 2023-07-12 18:43:19,364 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821/jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta 2023-07-12 18:43:19,364 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45039,DS-3d7629ec-da28-4d9a-b55e-6763e4437bb1,DISK], DatanodeInfoWithStorage[127.0.0.1:42053,DS-4ad92ce8-6d56-445f-9b36-b8468e44fc58,DISK]] 2023-07-12 18:43:19,365 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 18:43:19,367 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 18:43:19,381 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 18:43:19,418 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 18:43:19,422 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 18:43:19,422 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 18:43:19,422 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 18:43:19,422 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 18:43:19,426 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 18:43:19,428 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/info 2023-07-12 18:43:19,428 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/info 2023-07-12 18:43:19,429 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 18:43:19,430 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 18:43:19,431 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 18:43:19,432 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/rep_barrier 2023-07-12 18:43:19,432 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/rep_barrier 2023-07-12 18:43:19,433 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 18:43:19,434 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 18:43:19,434 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 18:43:19,435 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/table 2023-07-12 18:43:19,435 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/table 2023-07-12 18:43:19,435 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 18:43:19,436 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 18:43:19,438 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740 2023-07-12 18:43:19,442 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740 2023-07-12 18:43:19,445 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 18:43:19,448 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 18:43:19,449 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62070753, jitterRate=-0.07507370412349701}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 18:43:19,450 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 18:43:19,463 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689187399313 2023-07-12 18:43:19,481 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 18:43:19,482 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 18:43:19,483 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,39999,1689187397821, state=OPEN 2023-07-12 18:43:19,486 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 18:43:19,486 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 18:43:19,490 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 18:43:19,490 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,39999,1689187397821 in 353 msec 2023-07-12 18:43:19,495 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 18:43:19,495 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 555 msec 2023-07-12 18:43:19,500 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 783 msec 2023-07-12 18:43:19,500 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689187399500, completionTime=-1 2023-07-12 18:43:19,500 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-12 18:43:19,500 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 18:43:19,555 DEBUG [hconnection-0x4d387d5e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 18:43:19,558 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34182, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 18:43:19,574 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-12 18:43:19,574 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689187459574 2023-07-12 18:43:19,575 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689187519575 2023-07-12 18:43:19,575 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 74 msec 2023-07-12 18:43:19,604 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,33301,1689187396898-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:19,604 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,33301,1689187396898-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:19,604 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,33301,1689187396898-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:19,606 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:33301, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:19,606 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:19,613 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 18:43:19,620 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 18:43:19,622 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 18:43:19,630 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 18:43:19,633 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 18:43:19,636 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 18:43:19,657 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/.tmp/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:19,659 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/.tmp/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c empty. 2023-07-12 18:43:19,660 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/.tmp/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:19,661 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 18:43:19,699 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 18:43:19,701 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 01c6221c5c54d75c1882a9c5fc3b040c, NAME => 'hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/.tmp 2023-07-12 18:43:19,718 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 18:43:19,718 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 01c6221c5c54d75c1882a9c5fc3b040c, disabling compactions & flushes 2023-07-12 18:43:19,718 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:19,718 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:19,718 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. after waiting 0 ms 2023-07-12 18:43:19,718 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:19,718 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:19,718 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 01c6221c5c54d75c1882a9c5fc3b040c: 2023-07-12 18:43:19,722 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 18:43:19,737 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689187399725"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689187399725"}]},"ts":"1689187399725"} 2023-07-12 18:43:19,760 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 18:43:19,762 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 18:43:19,766 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689187399762"}]},"ts":"1689187399762"} 2023-07-12 18:43:19,770 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 18:43:19,774 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 18:43:19,775 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 18:43:19,775 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 18:43:19,775 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 18:43:19,777 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=01c6221c5c54d75c1882a9c5fc3b040c, ASSIGN}] 2023-07-12 18:43:19,781 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=01c6221c5c54d75c1882a9c5fc3b040c, ASSIGN 2023-07-12 18:43:19,783 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=01c6221c5c54d75c1882a9c5fc3b040c, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39999,1689187397821; forceNewPlan=false, retain=false 2023-07-12 18:43:19,937 INFO [jenkins-hbase20:33301] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 18:43:19,938 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=01c6221c5c54d75c1882a9c5fc3b040c, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:19,938 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689187399938"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689187399938"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689187399938"}]},"ts":"1689187399938"} 2023-07-12 18:43:19,946 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 01c6221c5c54d75c1882a9c5fc3b040c, server=jenkins-hbase20.apache.org,39999,1689187397821}] 2023-07-12 18:43:20,109 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:20,109 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 01c6221c5c54d75c1882a9c5fc3b040c, NAME => 'hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c.', STARTKEY => '', ENDKEY => ''} 2023-07-12 18:43:20,111 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:20,111 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 18:43:20,111 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:20,111 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:20,114 INFO [StoreOpener-01c6221c5c54d75c1882a9c5fc3b040c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:20,117 DEBUG [StoreOpener-01c6221c5c54d75c1882a9c5fc3b040c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/info 2023-07-12 18:43:20,117 DEBUG [StoreOpener-01c6221c5c54d75c1882a9c5fc3b040c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/info 2023-07-12 18:43:20,118 INFO [StoreOpener-01c6221c5c54d75c1882a9c5fc3b040c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 01c6221c5c54d75c1882a9c5fc3b040c columnFamilyName info 2023-07-12 18:43:20,119 INFO [StoreOpener-01c6221c5c54d75c1882a9c5fc3b040c-1] regionserver.HStore(310): Store=01c6221c5c54d75c1882a9c5fc3b040c/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 18:43:20,121 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:20,122 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:20,127 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:20,131 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 18:43:20,131 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 01c6221c5c54d75c1882a9c5fc3b040c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73878325, jitterRate=0.10087282955646515}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 18:43:20,131 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 01c6221c5c54d75c1882a9c5fc3b040c: 2023-07-12 18:43:20,133 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c., pid=6, masterSystemTime=1689187400100 2023-07-12 18:43:20,137 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:20,137 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:20,138 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=01c6221c5c54d75c1882a9c5fc3b040c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:20,139 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689187400138"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689187400138"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689187400138"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689187400138"}]},"ts":"1689187400138"} 2023-07-12 18:43:20,148 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 18:43:20,148 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 01c6221c5c54d75c1882a9c5fc3b040c, server=jenkins-hbase20.apache.org,39999,1689187397821 in 198 msec 2023-07-12 18:43:20,152 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 18:43:20,153 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=01c6221c5c54d75c1882a9c5fc3b040c, ASSIGN in 371 msec 2023-07-12 18:43:20,154 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 18:43:20,154 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689187400154"}]},"ts":"1689187400154"} 2023-07-12 18:43:20,157 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 18:43:20,160 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 18:43:20,162 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 537 msec 2023-07-12 18:43:20,234 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 18:43:20,236 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 18:43:20,236 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 18:43:20,279 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 18:43:20,294 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 18:43:20,301 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 31 msec 2023-07-12 18:43:20,312 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 18:43:20,323 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 18:43:20,327 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 14 msec 2023-07-12 18:43:20,343 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 18:43:20,345 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 18:43:20,346 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.417sec 2023-07-12 18:43:20,350 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 18:43:20,352 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 18:43:20,352 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 18:43:20,354 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,33301,1689187396898-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 18:43:20,354 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,33301,1689187396898-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 18:43:20,362 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 18:43:20,400 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ReadOnlyZKClient(139): Connect 0x0af4a250 to 127.0.0.1:63743 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 18:43:20,406 DEBUG [Listener at localhost.localdomain/46185] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70c57fa0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 18:43:20,422 DEBUG [hconnection-0x14efe23d-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 18:43:20,434 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34184, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 18:43:20,444 INFO [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,33301,1689187396898 2023-07-12 18:43:20,453 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 18:43:20,455 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:33672, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 18:43:20,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 18:43:20,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-12 18:43:20,471 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 18:43:20,473 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 18:43:20,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(700): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-12 18:43:20,476 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/.tmp/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:20,477 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/.tmp/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca empty. 2023-07-12 18:43:20,480 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/.tmp/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:20,480 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-12 18:43:20,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 18:43:20,503 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-12 18:43:20,504 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => f1f74138b310c7bba5d5da0cd13ec9ca, NAME => 't,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/.tmp 2023-07-12 18:43:20,521 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 18:43:20,521 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing f1f74138b310c7bba5d5da0cd13ec9ca, disabling compactions & flushes 2023-07-12 18:43:20,521 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:20,521 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:20,521 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. after waiting 0 ms 2023-07-12 18:43:20,521 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:20,521 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:20,521 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for f1f74138b310c7bba5d5da0cd13ec9ca: 2023-07-12 18:43:20,525 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 18:43:20,527 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689187400527"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689187400527"}]},"ts":"1689187400527"} 2023-07-12 18:43:20,530 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 18:43:20,532 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 18:43:20,532 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689187400532"}]},"ts":"1689187400532"} 2023-07-12 18:43:20,534 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-12 18:43:20,538 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-12 18:43:20,539 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 18:43:20,540 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 18:43:20,540 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 18:43:20,540 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f1f74138b310c7bba5d5da0cd13ec9ca, ASSIGN}] 2023-07-12 18:43:20,542 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f1f74138b310c7bba5d5da0cd13ec9ca, ASSIGN 2023-07-12 18:43:20,543 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=f1f74138b310c7bba5d5da0cd13ec9ca, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,46475,1689187397866; forceNewPlan=false, retain=false 2023-07-12 18:43:20,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 18:43:20,693 INFO [jenkins-hbase20:33301] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 18:43:20,695 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f1f74138b310c7bba5d5da0cd13ec9ca, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,46475,1689187397866 2023-07-12 18:43:20,696 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689187400695"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689187400695"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689187400695"}]},"ts":"1689187400695"} 2023-07-12 18:43:20,703 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure f1f74138b310c7bba5d5da0cd13ec9ca, server=jenkins-hbase20.apache.org,46475,1689187397866}] 2023-07-12 18:43:20,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 18:43:20,858 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,46475,1689187397866 2023-07-12 18:43:20,859 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 18:43:20,862 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43774, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 18:43:20,870 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:20,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f1f74138b310c7bba5d5da0cd13ec9ca, NAME => 't,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.', STARTKEY => '', ENDKEY => ''} 2023-07-12 18:43:20,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:20,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 18:43:20,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:20,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:20,874 INFO [StoreOpener-f1f74138b310c7bba5d5da0cd13ec9ca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:20,877 DEBUG [StoreOpener-f1f74138b310c7bba5d5da0cd13ec9ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/f 2023-07-12 18:43:20,878 DEBUG [StoreOpener-f1f74138b310c7bba5d5da0cd13ec9ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/f 2023-07-12 18:43:20,878 INFO [StoreOpener-f1f74138b310c7bba5d5da0cd13ec9ca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f1f74138b310c7bba5d5da0cd13ec9ca columnFamilyName f 2023-07-12 18:43:20,880 INFO [StoreOpener-f1f74138b310c7bba5d5da0cd13ec9ca-1] regionserver.HStore(310): Store=f1f74138b310c7bba5d5da0cd13ec9ca/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 18:43:20,882 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:20,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:20,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:20,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 18:43:20,897 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened f1f74138b310c7bba5d5da0cd13ec9ca; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66647916, jitterRate=-0.0068686604499816895}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 18:43:20,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for f1f74138b310c7bba5d5da0cd13ec9ca: 2023-07-12 18:43:20,898 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca., pid=11, masterSystemTime=1689187400858 2023-07-12 18:43:20,902 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:20,903 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:20,904 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f1f74138b310c7bba5d5da0cd13ec9ca, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,46475,1689187397866 2023-07-12 18:43:20,904 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689187400904"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689187400904"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689187400904"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689187400904"}]},"ts":"1689187400904"} 2023-07-12 18:43:20,910 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-12 18:43:20,910 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure f1f74138b310c7bba5d5da0cd13ec9ca, server=jenkins-hbase20.apache.org,46475,1689187397866 in 204 msec 2023-07-12 18:43:20,913 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-12 18:43:20,913 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=f1f74138b310c7bba5d5da0cd13ec9ca, ASSIGN in 370 msec 2023-07-12 18:43:20,914 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 18:43:20,914 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689187400914"}]},"ts":"1689187400914"} 2023-07-12 18:43:20,916 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-12 18:43:20,919 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 18:43:20,921 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 452 msec 2023-07-12 18:43:21,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 18:43:21,097 INFO [Listener at localhost.localdomain/46185] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-12 18:43:21,098 DEBUG [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-12 18:43:21,099 INFO [Listener at localhost.localdomain/46185] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 18:43:21,104 INFO [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-12 18:43:21,105 INFO [Listener at localhost.localdomain/46185] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 18:43:21,105 INFO [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-12 18:43:21,108 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 18:43:21,112 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43782, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 18:43:21,165 INFO [Listener at localhost.localdomain/46185] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=293, OpenFileDescriptor=585, MaxFileDescriptor=60000, SystemLoadAverage=145, ProcessCount=167, AvailableMemoryMB=6425 2023-07-12 18:43:21,167 INFO [Listener at localhost.localdomain/46185] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-12 18:43:21,173 DEBUG [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-12 18:43:21,174 DEBUG [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:21,174 INFO [Listener at localhost.localdomain/46185] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-12 18:43:21,174 INFO [Listener at localhost.localdomain/46185] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-12 18:43:21,180 INFO [Listener at localhost.localdomain/46185] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689187400154/Put/vlen=2/seqid=0} 2023-07-12 18:43:21,180 INFO [Listener at localhost.localdomain/46185] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-12 18:43:21,180 INFO [Listener at localhost.localdomain/46185] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c./info:regioninfo/1689187400138/Put/vlen=41/seqid=0, hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c./info:seqnumDuringOpen/1689187400138/Put/vlen=8/seqid=0, hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c./info:server/1689187400138/Put/vlen=32/seqid=0, hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c./info:serverstartcode/1689187400138/Put/vlen=8/seqid=0, hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c./info:sn/1689187399938/Put/vlen=46/seqid=0, hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c./info:state/1689187400138/Put/vlen=4/seqid=0} 2023-07-12 18:43:21,181 INFO [Listener at localhost.localdomain/46185] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 01c6221c5c54d75c1882a9c5fc3b040c, NAME => 'hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c.', STARTKEY => '', ENDKEY => ''} 2023-07-12 18:43:21,181 INFO [Listener at localhost.localdomain/46185] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689187400914/Put/vlen=2/seqid=0} 2023-07-12 18:43:21,181 INFO [Listener at localhost.localdomain/46185] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-12 18:43:21,181 INFO [Listener at localhost.localdomain/46185] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca./info:regioninfo/1689187400904/Put/vlen=35/seqid=0, t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca./info:seqnumDuringOpen/1689187400904/Put/vlen=8/seqid=0, t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca./info:server/1689187400904/Put/vlen=32/seqid=0, t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca./info:serverstartcode/1689187400904/Put/vlen=8/seqid=0, t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca./info:sn/1689187400695/Put/vlen=46/seqid=0, t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca./info:state/1689187400904/Put/vlen=4/seqid=0} 2023-07-12 18:43:21,181 INFO [Listener at localhost.localdomain/46185] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => f1f74138b310c7bba5d5da0cd13ec9ca, NAME => 't,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.', STARTKEY => '', ENDKEY => ''} 2023-07-12 18:43:21,182 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ReadOnlyZKClient(139): Connect 0x1717644b to 127.0.0.1:63743 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 18:43:21,187 DEBUG [Listener at localhost.localdomain/46185] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@137695c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 18:43:21,187 INFO [Listener at localhost.localdomain/46185] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-12 18:43:21,187 INFO [Listener at localhost.localdomain/46185] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-12 18:43:21,191 DEBUG [hconnection-0x3301b991-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 18:43:21,198 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34196, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 18:43:21,205 DEBUG [hconnection-0x3301b991-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 18:43:21,209 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43784, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 18:43:21,212 INFO [Listener at localhost.localdomain/46185] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-12 18:43:21,216 ERROR [Listener at localhost.localdomain/46185] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,46475,1689187397866: die! ***** 2023-07-12 18:43:21,217 ERROR [Listener at localhost.localdomain/46185] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 18:43:21,220 DEBUG [Listener at localhost.localdomain/46185] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 18:43:21,224 DEBUG [Listener at localhost.localdomain/46185] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 18:43:21,224 DEBUG [Listener at localhost.localdomain/46185] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 18:43:21,225 DEBUG [Listener at localhost.localdomain/46185] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 18:43:21,227 INFO [Listener at localhost.localdomain/46185] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 912261120, "init": 524288000, "max": 2051014656, "used": 409208592 }, "NonHeapMemoryUsage": { "committed": 113008640, "init": 2555904, "max": -1, "used": 111112232 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org" } ] } 2023-07-12 18:43:21,231 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33301] master.MasterRpcServices(612): jenkins-hbase20.apache.org,46475,1689187397866 reported a fatal error: ***** ABORTING region server jenkins-hbase20.apache.org,46475,1689187397866: die! ***** 2023-07-12 18:43:21,234 INFO [Listener at localhost.localdomain/46185] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,46475,1689187397866' ***** 2023-07-12 18:43:21,235 INFO [Listener at localhost.localdomain/46185] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-12 18:43:21,235 INFO [RS:1;jenkins-hbase20:46475] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 18:43:21,235 INFO [RS:1;jenkins-hbase20:46475] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 18:43:21,235 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 18:43:21,235 INFO [RS:1;jenkins-hbase20:46475] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 18:43:21,236 INFO [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(3305): Received CLOSE for f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:21,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46475] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner -7216532152433770495 2023-07-12 18:43:21,236 INFO [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,46475,1689187397866 2023-07-12 18:43:21,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46475] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,46475,1689187397866 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 18:43:21,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing f1f74138b310c7bba5d5da0cd13ec9ca, disabling compactions & flushes 2023-07-12 18:43:21,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46475] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 148.251.75.209:43784 deadline: 1689187461236, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,46475,1689187397866 aborting 2023-07-12 18:43:21,237 DEBUG [RS:1;jenkins-hbase20:46475] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x19b6a5aa to 127.0.0.1:63743 2023-07-12 18:43:21,237 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:21,238 DEBUG [RS:1;jenkins-hbase20:46475] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 18:43:21,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:21,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. after waiting 0 ms 2023-07-12 18:43:21,238 INFO [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-12 18:43:21,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:21,238 DEBUG [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(1478): Online Regions={f1f74138b310c7bba5d5da0cd13ec9ca=t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.} 2023-07-12 18:43:21,238 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:21,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:21,239 DEBUG [Listener at localhost.localdomain/46185] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause 2023-07-12 18:43:21,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for f1f74138b310c7bba5d5da0cd13ec9ca: org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 18:43:21,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,46475,1689187397866 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,46475,1689187397866 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-12 18:43:21,242 DEBUG [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(1504): Waiting on 2023-07-12 18:43:21,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46475] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,46475,1689187397866 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 18:43:21,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46475] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 148.251.75.209:43784 deadline: 1689187461243, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,46475,1689187397866 aborting 2023-07-12 18:43:21,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46475] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,46475,1689187397866 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 18:43:21,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46475] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Scan size: 125 connection: 148.251.75.209:43784 deadline: 1689187461355, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,46475,1689187397866 aborting 2023-07-12 18:43:21,442 INFO [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,46475,1689187397866; all regions closed. 2023-07-12 18:43:21,457 DEBUG [RS:1;jenkins-hbase20:46475] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 18:43:21,458 INFO [RS:1;jenkins-hbase20:46475] regionserver.LeaseManager(133): Closed leases 2023-07-12 18:43:21,458 INFO [RS:1;jenkins-hbase20:46475] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-12 18:43:21,458 INFO [RS:1;jenkins-hbase20:46475] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 18:43:21,458 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 18:43:21,458 INFO [RS:1;jenkins-hbase20:46475] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 18:43:21,459 INFO [RS:1;jenkins-hbase20:46475] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 18:43:21,459 INFO [RS:1;jenkins-hbase20:46475] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:46475 2023-07-12 18:43:21,468 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 18:43:21,468 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:46475-0x100827ea6a00002, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,46475,1689187397866 2023-07-12 18:43:21,468 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,46475,1689187397866 2023-07-12 18:43:21,469 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:46475-0x100827ea6a00002, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 18:43:21,469 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 18:43:21,470 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,46475,1689187397866] 2023-07-12 18:43:21,470 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,46475,1689187397866; numProcessing=1 2023-07-12 18:43:21,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:21,471 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase20.apache.org,46475,1689187397866 znode expired, triggering replicatorRemoved event 2023-07-12 18:43:21,471 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,46475,1689187397866 already deleted, retry=false 2023-07-12 18:43:21,471 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase20.apache.org,46475,1689187397866 on jenkins-hbase20.apache.org,33301,1689187396898 2023-07-12 18:43:21,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:21,478 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase20.apache.org,46475,1689187397866, splitWal=true, meta=false 2023-07-12 18:43:21,479 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase20.apache.org,46475,1689187397866 (carryingMeta=false) jenkins-hbase20.apache.org,46475,1689187397866/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@2bac6002[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-12 18:43:21,482 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,46475,1689187397866, splitWal=true, meta=false 2023-07-12 18:43:21,483 INFO [PEWorker-1] procedure.ServerCrashProcedure(199): jenkins-hbase20.apache.org,46475,1689187397866 had 1 regions 2023-07-12 18:43:21,485 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,46475,1689187397866, splitWal=true, meta=false, isMeta: false 2023-07-12 18:43:21,488 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866-splitting 2023-07-12 18:43:21,490 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase20.apache.org,46475,1689187397866 WAL count=1, meta=false 2023-07-12 18:43:21,493 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133}] 2023-07-12 18:43:21,529 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:21,532 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133, worker=jenkins-hbase20.apache.org,39999,1689187397821}] 2023-07-12 18:43:21,563 WARN [RPCClient-NioEventLoopGroup-5-5] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:46475 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:46475 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 18:43:21,564 DEBUG [RPCClient-NioEventLoopGroup-5-5] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:46475 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:46475 2023-07-12 18:43:21,571 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:46475-0x100827ea6a00002, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 18:43:21,571 INFO [RS:1;jenkins-hbase20:46475] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,46475,1689187397866; zookeeper connection closed. 2023-07-12 18:43:21,571 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:46475-0x100827ea6a00002, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 18:43:21,571 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@14d4fa50] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@14d4fa50 2023-07-12 18:43:21,709 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39999] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-12 18:43:21,727 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866-splitting/jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133, size=1.1 K (1149bytes) 2023-07-12 18:43:21,727 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866-splitting/jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133 2023-07-12 18:43:21,731 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866-splitting/jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133 after 4ms 2023-07-12 18:43:21,736 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(311): Open hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866-splitting/jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133 took 10ms 2023-07-12 18:43:21,749 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(346): Last flushed sequenceid for f1f74138b310c7bba5d5da0cd13ec9ca: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-12 18:43:21,751 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866-splitting/jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133 so closing down 2023-07-12 18:43:21,751 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-12 18:43:21,756 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-12 18:43:21,759 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133.temp 2023-07-12 18:43:21,760 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-12 18:43:21,776 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-12 18:43:21,779 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133.temp to hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/recovered.edits/0000000000000000013 2023-07-12 18:43:21,779 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 42 ms; skipped=1; WAL=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866-splitting/jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133, size=1.1 K, length=1149, corrupted=false, cancelled=false 2023-07-12 18:43:21,780 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866-splitting/jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133, journal: Splitting hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866-splitting/jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133, size=1.1 K (1149bytes) at 1689187401727Finishing writing output for hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866-splitting/jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133 so closing down at 1689187401751 (+24 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133.temp at 1689187401759 (+8 ms)3 split writer threads finished at 1689187401760 (+1 ms)Closed recovered edits writer path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689187401776 (+16 ms)Rename recovered edits hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133.temp to hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/recovered.edits/0000000000000000013 at 1689187401779 (+3 ms)Processed 11 edits across 1 Regions in 42 ms; skipped=1; WAL=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866-splitting/jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133, size=1.1 K, length=1149, corrupted=false, cancelled=false at 1689187401780 (+1 ms) 2023-07-12 18:43:21,780 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866-splitting/jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133 2023-07-12 18:43:21,783 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-12 18:43:21,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster(3641): Remote procedure done, pid=14 2023-07-12 18:43:21,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866-splitting/jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133 to hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/oldWALs 2023-07-12 18:43:21,799 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-12 18:43:21,800 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133, worker=jenkins-hbase20.apache.org,39999,1689187397821 in 263 msec 2023-07-12 18:43:21,801 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:21,805 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-12 18:43:21,805 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase20.apache.org%2C46475%2C1689187397866.1689187399133, worker=jenkins-hbase20.apache.org,39999,1689187397821 in 310 msec 2023-07-12 18:43:21,808 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866-splitting dir is empty, no logs to split. 2023-07-12 18:43:21,808 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase20.apache.org,46475,1689187397866 WAL count=0, meta=false 2023-07-12 18:43:21,808 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase20.apache.org,46475,1689187397866 WAL splitting is done? wals=0, meta=false 2023-07-12 18:43:21,810 INFO [PEWorker-1] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase20.apache.org,46475,1689187397866 failed, ignore...File hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,46475,1689187397866-splitting does not exist. 2023-07-12 18:43:21,812 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f1f74138b310c7bba5d5da0cd13ec9ca, ASSIGN}] 2023-07-12 18:43:21,813 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f1f74138b310c7bba5d5da0cd13ec9ca, ASSIGN 2023-07-12 18:43:21,814 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=f1f74138b310c7bba5d5da0cd13ec9ca, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 18:43:21,877 DEBUG [hconnection-0x3301b991-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:46475 this server is in the failed servers list 2023-07-12 18:43:21,965 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=f1f74138b310c7bba5d5da0cd13ec9ca, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:21,965 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689187401965"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689187401965"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689187401965"}]},"ts":"1689187401965"} 2023-07-12 18:43:21,968 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure f1f74138b310c7bba5d5da0cd13ec9ca, server=jenkins-hbase20.apache.org,39999,1689187397821}] 2023-07-12 18:43:22,003 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 18:43:22,128 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:22,128 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f1f74138b310c7bba5d5da0cd13ec9ca, NAME => 't,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.', STARTKEY => '', ENDKEY => ''} 2023-07-12 18:43:22,128 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:22,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 18:43:22,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:22,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:22,132 INFO [StoreOpener-f1f74138b310c7bba5d5da0cd13ec9ca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:22,134 DEBUG [StoreOpener-f1f74138b310c7bba5d5da0cd13ec9ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/f 2023-07-12 18:43:22,134 DEBUG [StoreOpener-f1f74138b310c7bba5d5da0cd13ec9ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/f 2023-07-12 18:43:22,135 INFO [StoreOpener-f1f74138b310c7bba5d5da0cd13ec9ca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f1f74138b310c7bba5d5da0cd13ec9ca columnFamilyName f 2023-07-12 18:43:22,136 INFO [StoreOpener-f1f74138b310c7bba5d5da0cd13ec9ca-1] regionserver.HStore(310): Store=f1f74138b310c7bba5d5da0cd13ec9ca/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 18:43:22,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:22,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:22,145 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/recovered.edits/0000000000000000013 2023-07-12 18:43:22,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/recovered.edits/0000000000000000013 2023-07-12 18:43:22,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing f1f74138b310c7bba5d5da0cd13ec9ca 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-12 18:43:22,222 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/.tmp/f/7fd593e96e85408da538ffb739043cbe 2023-07-12 18:43:22,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/.tmp/f/7fd593e96e85408da538ffb739043cbe as hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/f/7fd593e96e85408da538ffb739043cbe 2023-07-12 18:43:22,271 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/f/7fd593e96e85408da538ffb739043cbe, entries=10, sequenceid=13, filesize=4.8 K 2023-07-12 18:43:22,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for f1f74138b310c7bba5d5da0cd13ec9ca in 118ms, sequenceid=13, compaction requested=false; wal=null 2023-07-12 18:43:22,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-12 18:43:22,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/recovered.edits/0000000000000000013 2023-07-12 18:43:22,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:22,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-12 18:43:22,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened f1f74138b310c7bba5d5da0cd13ec9ca; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70815354, jitterRate=0.0552310049533844}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 18:43:22,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for f1f74138b310c7bba5d5da0cd13ec9ca: 2023-07-12 18:43:22,288 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca., pid=16, masterSystemTime=1689187402121 2023-07-12 18:43:22,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:22,290 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:22,291 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=f1f74138b310c7bba5d5da0cd13ec9ca, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:22,292 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689187402291"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689187402291"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689187402291"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689187402291"}]},"ts":"1689187402291"} 2023-07-12 18:43:22,299 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-12 18:43:22,299 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure f1f74138b310c7bba5d5da0cd13ec9ca, server=jenkins-hbase20.apache.org,39999,1689187397821 in 327 msec 2023-07-12 18:43:22,302 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-12 18:43:22,302 INFO [PEWorker-1] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase20.apache.org,46475,1689187397866 after splitting done 2023-07-12 18:43:22,302 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=f1f74138b310c7bba5d5da0cd13ec9ca, ASSIGN in 487 msec 2023-07-12 18:43:22,302 DEBUG [PEWorker-1] master.DeadServer(114): Removed jenkins-hbase20.apache.org,46475,1689187397866 from processing; numProcessing=0 2023-07-12 18:43:22,305 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase20.apache.org,46475,1689187397866, splitWal=true, meta=false in 829 msec 2023-07-12 18:43:22,406 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1717644b to 127.0.0.1:63743 2023-07-12 18:43:22,407 DEBUG [Listener at localhost.localdomain/46185] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 18:43:22,407 INFO [Listener at localhost.localdomain/46185] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-12 18:43:22,421 INFO [Listener at localhost.localdomain/46185] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=270 (was 293), OpenFileDescriptor=567 (was 585), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=145 (was 145), ProcessCount=167 (was 167), AvailableMemoryMB=6371 (was 6425) 2023-07-12 18:43:22,431 INFO [Listener at localhost.localdomain/46185] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=270, OpenFileDescriptor=567, MaxFileDescriptor=60000, SystemLoadAverage=145, ProcessCount=167, AvailableMemoryMB=6370 2023-07-12 18:43:22,432 INFO [Listener at localhost.localdomain/46185] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase20:46475 2023-07-12 18:43:22,445 INFO [Listener at localhost.localdomain/46185] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-12 18:43:22,445 INFO [Listener at localhost.localdomain/46185] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 18:43:22,445 INFO [Listener at localhost.localdomain/46185] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 18:43:22,445 INFO [Listener at localhost.localdomain/46185] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 18:43:22,446 INFO [Listener at localhost.localdomain/46185] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 18:43:22,446 INFO [Listener at localhost.localdomain/46185] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 18:43:22,446 INFO [Listener at localhost.localdomain/46185] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 18:43:22,447 INFO [Listener at localhost.localdomain/46185] ipc.NettyRpcServer(120): Bind to /148.251.75.209:41053 2023-07-12 18:43:22,448 INFO [Listener at localhost.localdomain/46185] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 18:43:22,449 DEBUG [Listener at localhost.localdomain/46185] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 18:43:22,449 INFO [Listener at localhost.localdomain/46185] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 18:43:22,451 INFO [Listener at localhost.localdomain/46185] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 18:43:22,452 INFO [Listener at localhost.localdomain/46185] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41053 connecting to ZooKeeper ensemble=127.0.0.1:63743 2023-07-12 18:43:22,456 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:410530x0, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 18:43:22,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:41053-0x100827ea6a00008 connected 2023-07-12 18:43:22,458 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ZKUtil(162): regionserver:41053-0x100827ea6a00008, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 18:43:22,459 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ZKUtil(162): regionserver:41053-0x100827ea6a00008, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-12 18:43:22,460 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ZKUtil(164): regionserver:41053-0x100827ea6a00008, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 18:43:22,460 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41053 2023-07-12 18:43:22,461 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41053 2023-07-12 18:43:22,461 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41053 2023-07-12 18:43:22,462 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41053 2023-07-12 18:43:22,462 DEBUG [Listener at localhost.localdomain/46185] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41053 2023-07-12 18:43:22,465 INFO [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(951): ClusterId : abb7fc94-1c96-4221-9e93-46bbae4759c5 2023-07-12 18:43:22,466 DEBUG [RS:2;jenkins-hbase20:41053] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 18:43:22,468 DEBUG [RS:2;jenkins-hbase20:41053] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 18:43:22,468 DEBUG [RS:2;jenkins-hbase20:41053] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 18:43:22,470 DEBUG [RS:2;jenkins-hbase20:41053] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 18:43:22,472 DEBUG [RS:2;jenkins-hbase20:41053] zookeeper.ReadOnlyZKClient(139): Connect 0x308c5166 to 127.0.0.1:63743 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 18:43:22,476 DEBUG [RS:2;jenkins-hbase20:41053] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e4ba907, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 18:43:22,477 DEBUG [RS:2;jenkins-hbase20:41053] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a743a16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 18:43:22,487 DEBUG [RS:2;jenkins-hbase20:41053] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase20:41053 2023-07-12 18:43:22,487 INFO [RS:2;jenkins-hbase20:41053] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 18:43:22,487 INFO [RS:2;jenkins-hbase20:41053] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 18:43:22,487 DEBUG [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 18:43:22,488 INFO [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,33301,1689187396898 with isa=jenkins-hbase20.apache.org/148.251.75.209:41053, startcode=1689187402445 2023-07-12 18:43:22,488 DEBUG [RS:2;jenkins-hbase20:41053] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 18:43:22,492 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39731, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 18:43:22,493 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33301] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:22,493 DEBUG [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d 2023-07-12 18:43:22,494 DEBUG [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:33493 2023-07-12 18:43:22,494 DEBUG [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 18:43:22,495 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 18:43:22,495 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 18:43:22,495 DEBUG [RS:2;jenkins-hbase20:41053] zookeeper.ZKUtil(162): regionserver:41053-0x100827ea6a00008, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:22,495 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,41053,1689187402445] 2023-07-12 18:43:22,496 WARN [RS:2;jenkins-hbase20:41053] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 18:43:22,496 INFO [RS:2;jenkins-hbase20:41053] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 18:43:22,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:22,497 DEBUG [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:22,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:22,502 DEBUG [RS:2;jenkins-hbase20:41053] zookeeper.ZKUtil(162): regionserver:41053-0x100827ea6a00008, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:22,503 DEBUG [RS:2;jenkins-hbase20:41053] zookeeper.ZKUtil(162): regionserver:41053-0x100827ea6a00008, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:22,504 DEBUG [RS:2;jenkins-hbase20:41053] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 18:43:22,504 INFO [RS:2;jenkins-hbase20:41053] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 18:43:22,506 INFO [RS:2;jenkins-hbase20:41053] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 18:43:22,507 INFO [RS:2;jenkins-hbase20:41053] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 18:43:22,507 INFO [RS:2;jenkins-hbase20:41053] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:22,508 INFO [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 18:43:22,509 INFO [RS:2;jenkins-hbase20:41053] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:22,510 DEBUG [RS:2;jenkins-hbase20:41053] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:22,510 DEBUG [RS:2;jenkins-hbase20:41053] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:22,510 DEBUG [RS:2;jenkins-hbase20:41053] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:22,510 DEBUG [RS:2;jenkins-hbase20:41053] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:22,510 DEBUG [RS:2;jenkins-hbase20:41053] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:22,510 DEBUG [RS:2;jenkins-hbase20:41053] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-12 18:43:22,510 DEBUG [RS:2;jenkins-hbase20:41053] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:22,510 DEBUG [RS:2;jenkins-hbase20:41053] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:22,510 DEBUG [RS:2;jenkins-hbase20:41053] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:22,510 DEBUG [RS:2;jenkins-hbase20:41053] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-12 18:43:22,511 INFO [RS:2;jenkins-hbase20:41053] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:22,512 INFO [RS:2;jenkins-hbase20:41053] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:22,512 INFO [RS:2;jenkins-hbase20:41053] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:22,521 INFO [RS:2;jenkins-hbase20:41053] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 18:43:22,521 INFO [RS:2;jenkins-hbase20:41053] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,41053,1689187402445-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 18:43:22,530 INFO [RS:2;jenkins-hbase20:41053] regionserver.Replication(203): jenkins-hbase20.apache.org,41053,1689187402445 started 2023-07-12 18:43:22,530 INFO [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,41053,1689187402445, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:41053, sessionid=0x100827ea6a00008 2023-07-12 18:43:22,530 INFO [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase20:41053,5,FailOnTimeoutGroup] 2023-07-12 18:43:22,530 DEBUG [RS:2;jenkins-hbase20:41053] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 18:43:22,530 INFO [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase20.apache.org,46475,1689187397866 2023-07-12 18:43:22,530 DEBUG [RS:2;jenkins-hbase20:41053] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:22,530 INFO [Listener at localhost.localdomain/46185] client.TestScannerTimeout(177): START ************ test3686b 2023-07-12 18:43:22,530 DEBUG [RS:2;jenkins-hbase20:41053] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,41053,1689187402445' 2023-07-12 18:43:22,530 DEBUG [RS:2;jenkins-hbase20:41053] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 18:43:22,531 DEBUG [RS:2;jenkins-hbase20:41053] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 18:43:22,531 DEBUG [RS:2;jenkins-hbase20:41053] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 18:43:22,531 DEBUG [RS:2;jenkins-hbase20:41053] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 18:43:22,531 DEBUG [RS:2;jenkins-hbase20:41053] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:22,531 DEBUG [RS:2;jenkins-hbase20:41053] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,41053,1689187402445' 2023-07-12 18:43:22,531 DEBUG [RS:2;jenkins-hbase20:41053] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 18:43:22,532 DEBUG [RS:2;jenkins-hbase20:41053] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 18:43:22,532 DEBUG [RS:2;jenkins-hbase20:41053] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 18:43:22,532 INFO [RS:2;jenkins-hbase20:41053] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 18:43:22,532 INFO [RS:2;jenkins-hbase20:41053] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 18:43:22,533 DEBUG [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-12 18:43:22,533 DEBUG [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:22,534 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ReadOnlyZKClient(139): Connect 0x3ab5710d to 127.0.0.1:63743 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 18:43:22,538 DEBUG [Listener at localhost.localdomain/46185] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@290422a6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 18:43:22,541 DEBUG [hconnection-0x315b0cc4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 18:43:22,543 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34202, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 18:43:22,640 INFO [RS:2;jenkins-hbase20:41053] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C41053%2C1689187402445, suffix=, logDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,41053,1689187402445, archiveDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/oldWALs, maxLogs=32 2023-07-12 18:43:22,661 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45039,DS-3d7629ec-da28-4d9a-b55e-6763e4437bb1,DISK] 2023-07-12 18:43:22,662 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42053,DS-4ad92ce8-6d56-445f-9b36-b8468e44fc58,DISK] 2023-07-12 18:43:22,665 INFO [RS:2;jenkins-hbase20:41053] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,41053,1689187402445/jenkins-hbase20.apache.org%2C41053%2C1689187402445.1689187402644 2023-07-12 18:43:22,665 DEBUG [RS:2;jenkins-hbase20:41053] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45039,DS-3d7629ec-da28-4d9a-b55e-6763e4437bb1,DISK], DatanodeInfoWithStorage[127.0.0.1:42053,DS-4ad92ce8-6d56-445f-9b36-b8468e44fc58,DISK]] 2023-07-12 18:43:24,941 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 18:43:25,006 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 18:43:25,007 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 18:43:27,024 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 18:43:27,025 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-12 18:43:37,548 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease 2301560175200305171 expired clientIPAndPort=148.251.75.209:34202, userName=jenkins, regionInfo=t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:39,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39999] regionserver.RSRpcServices(3094): Client tried to access missing scanner 2301560175200305171 2023-07-12 18:43:39,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39999] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 148.251.75.209:34202 deadline: 1689187479551, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '2301560175200305171'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-12 18:43:39,554 DEBUG [Listener at localhost.localdomain/46185] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '2301560175200305171'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '2301560175200305171'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '2301560175200305171'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-12 18:43:39,559 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ab5710d to 127.0.0.1:63743 2023-07-12 18:43:39,560 DEBUG [Listener at localhost.localdomain/46185] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 18:43:39,560 INFO [Listener at localhost.localdomain/46185] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-12 18:43:39,579 INFO [Listener at localhost.localdomain/46185] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=293 (was 270) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d-prefix:jenkins-hbase20.apache.org,41053,1689187402445 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2135704071_17 at /127.0.0.1:57000 [Receiving block BP-775451423-148.251.75.209-1689187394261:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=41053 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: jenkins-hbase20:41053Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=41053 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase20:41053 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=41053 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41053 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-775451423-148.251.75.209-1689187394261:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=41053 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41053 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-775451423-148.251.75.209-1689187394261:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=41053 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/cluster_55ffc31b-bad0-f043-ae3b-8971827a1352/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2135704071_17 at /127.0.0.1:53226 [Receiving block BP-775451423-148.251.75.209-1689187394261:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=41053 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=41053 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63743@0x3ab5710d-SendThread(127.0.0.1:63743) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost.localdomain:33493 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase20:41053-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41053 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63743@0x308c5166-SendThread(127.0.0.1:63743) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/cluster_55ffc31b-bad0-f043-ae3b-8971827a1352/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63743@0x308c5166 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/665307710.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63743@0x3ab5710d-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63743@0x3ab5710d java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/665307710.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63743@0x308c5166-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) - Thread LEAK? -, OpenFileDescriptor=571 (was 567) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=167 (was 145) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=6656 (was 6370) - AvailableMemoryMB LEAK? - 2023-07-12 18:43:39,592 INFO [Listener at localhost.localdomain/46185] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=293, OpenFileDescriptor=571, MaxFileDescriptor=60000, SystemLoadAverage=167, ProcessCount=167, AvailableMemoryMB=6655 2023-07-12 18:43:39,592 INFO [Listener at localhost.localdomain/46185] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase20:46475 2023-07-12 18:43:39,593 INFO [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase20.apache.org,46475,1689187397866 2023-07-12 18:43:39,593 INFO [Listener at localhost.localdomain/46185] client.TestScannerTimeout(104): START************ test2772 2023-07-12 18:43:39,599 DEBUG [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-12 18:43:39,599 DEBUG [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:39,600 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ReadOnlyZKClient(139): Connect 0x53953132 to 127.0.0.1:63743 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 18:43:39,609 DEBUG [Listener at localhost.localdomain/46185] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6db7cf1d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 18:43:39,609 ERROR [Listener at localhost.localdomain/46185] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,39999,1689187397821: die! ***** 2023-07-12 18:43:39,609 ERROR [Listener at localhost.localdomain/46185] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 18:43:39,609 DEBUG [Listener at localhost.localdomain/46185] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-12 18:43:39,610 DEBUG [Listener at localhost.localdomain/46185] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-12 18:43:39,618 DEBUG [Listener at localhost.localdomain/46185] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-12 18:43:39,620 DEBUG [Listener at localhost.localdomain/46185] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-12 18:43:39,640 INFO [Listener at localhost.localdomain/46185] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1011351552, "init": 524288000, "max": 2051014656, "used": 250423600 }, "NonHeapMemoryUsage": { "committed": 121856000, "init": 2555904, "max": -1, "used": 119616432 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 3, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 11985, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 61, "ProcessCallTime_min": 1, "ProcessCallTime_max": 2, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 2, "ProcessCallTime_75th_percentile": 2, "ProcessCallTime_90th_percentile": 2, "ProcessCallTime_95th_percentile": 2, "ProcessCallTime_98th_percentile": 2, "ProcessCallTime_99th_percentile": 2, "ProcessCallTime_99.9th_percentile": 2, "ProcessCallTime_TimeRangeCount_0-1": 1, "ProcessCallTime_TimeRangeCount_1-3": 1, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 61, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 61, "TotalCallTime_min": 1, "TotalCallTime_max": 2, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 2, "TotalCallTime_75th_percentile": 2, "TotalCallTime_90th_percentile": 2, "TotalCallTime_95th_percentile": 2, "TotalCallTime_98th_percentile": 2, "TotalCallTime_99th_percentile": 2, "TotalCallTime_99.9th_percentile": 2, "TotalCallTime_TimeRangeCount_0-1": 1, "TotalCallTime_TimeRangeCount_1-3": 1, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 61, "ResponseSize_min": 196, "ResponseSize_max": 680, "ResponseSize_mean": 438, "ResponseSize_25th_percentile": 196, "ResponseSize_median": 197, "ResponseSize_75th_percentile": 677, "ResponseSize_90th_percentile": 679, "ResponseSize_95th_percentile": 679, "ResponseSize_98th_percentile": 679, "ResponseSize_99th_percentile": 679, "ResponseSize_99.9th_percentile": 679, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 4, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 61, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16697 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:63743", "tag.serverName": "jenkins-hbase20.apache.org,39999,1689187397821", "tag.clusterId": "abb7fc94-1c96-4221-9e93-46bbae4759c5", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 1, "maxStoreFileCount": 1, "memStoreSize": 3312, "storeFileSize": 4926, "maxStoreFileAge": 16692, "minStoreFileAge": 16692, "avgStoreFileAge": 16692, "numReferenceFiles": 0, "regionServerStartTime": 1689187397821, "averageRegionSize": 2746, "storeFileIndexSize": 320, "staticIndexSize": 31, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 546, "localBytesRead": 546, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803016, "blockCacheCount": 1, "blockCacheDataBlockCount": 1, "blockCacheSize": 602872, "blockCacheCountHitPercent": 66.66666666666666, "blockCacheExpressHitPercent": 66.66666666666666, "l1CacheSize": 602872, "l1CacheFreeSize": 819803016, "l1CacheCount": 1, "l1CacheEvictionCount": 0, "l1CacheHitCount": 2, "l1CacheMissCount": 1, "l1CacheHitRatio": 0.6666666666666666, "l1CacheMissRatio": 0.3333333333333333, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 1640, "ByteBuffAllocatorPoolAllocationBytes": 1863680, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 46, "totalRowActionRequestCount": 57, "readRequestCount": 43, "filteredReadRequestCount": 0, "writeRequestCount": 14, "rpcGetRequestCount": 10, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 22, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 14, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 2, "blockCacheHitCountPrimary": 2, "blockCacheHitCachingCount": 2, "blockCacheMissCount": 1, "blockCacheMissCountPrimary": 1, "blockCacheMissCachingCount": 1, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 1, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 2, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 10, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 270, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 21, "ScanTime_min": 0, "ScanTime_max": 1, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 14, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 1, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 10, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 4926, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 1, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 21, "ScanSize_min": 155, "ScanSize_max": 655, "ScanSize_mean": 405, "ScanSize_25th_percentile": 156, "ScanSize_median": 157, "ScanSize_75th_percentile": 654, "ScanSize_90th_percentile": 654, "ScanSize_95th_percentile": 654, "ScanSize_98th_percentile": 654, "ScanSize_99th_percentile": 654, "ScanSize_99.9th_percentile": 654, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 270, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 1, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.6082023083132935, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 24, "ServerWriteQueryPerSecond_mean_rate": 1.159201547291097, "ServerWriteQueryPerSecond_1min_rate": 3.738243758742744, "ServerWriteQueryPerSecond_5min_rate": 4.565901237603426, "ServerWriteQueryPerSecond_15min_rate": 4.720662978343764 } ] } 2023-07-12 18:43:39,642 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33301] master.MasterRpcServices(612): jenkins-hbase20.apache.org,39999,1689187397821 reported a fatal error: ***** ABORTING region server jenkins-hbase20.apache.org,39999,1689187397821: die! ***** 2023-07-12 18:43:39,647 INFO [Listener at localhost.localdomain/46185] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,39999,1689187397821' ***** 2023-07-12 18:43:39,647 INFO [Listener at localhost.localdomain/46185] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-12 18:43:39,648 INFO [RS:0;jenkins-hbase20:39999] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 18:43:39,648 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 18:43:39,648 INFO [RS:0;jenkins-hbase20:39999] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-12 18:43:39,648 INFO [RS:0;jenkins-hbase20:39999] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-12 18:43:39,648 INFO [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(3305): Received CLOSE for f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:39,650 INFO [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(3305): Received CLOSE for 01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:39,651 INFO [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:39,651 DEBUG [RS:0;jenkins-hbase20:39999] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7310b462 to 127.0.0.1:63743 2023-07-12 18:43:39,651 DEBUG [RS:0;jenkins-hbase20:39999] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 18:43:39,651 INFO [RS:0;jenkins-hbase20:39999] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 18:43:39,651 INFO [RS:0;jenkins-hbase20:39999] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 18:43:39,651 INFO [RS:0;jenkins-hbase20:39999] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 18:43:39,651 INFO [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 18:43:39,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing f1f74138b310c7bba5d5da0cd13ec9ca, disabling compactions & flushes 2023-07-12 18:43:39,653 INFO [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-12 18:43:39,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:39,654 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 18:43:39,654 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:39,654 DEBUG [hconnection-0xdab2b0a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 18:43:39,653 DEBUG [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(1478): Online Regions={f1f74138b310c7bba5d5da0cd13ec9ca=t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca., 1588230740=hbase:meta,,1.1588230740, 01c6221c5c54d75c1882a9c5fc3b040c=hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c.} 2023-07-12 18:43:39,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. after waiting 0 ms 2023-07-12 18:43:39,654 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 18:43:39,655 DEBUG [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(1504): Waiting on 01c6221c5c54d75c1882a9c5fc3b040c, 1588230740, f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:39,655 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 18:43:39,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:39,655 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 18:43:39,655 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 18:43:39,656 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 3234 in region hbase:meta,,1.1588230740 2023-07-12 18:43:39,657 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 18:43:39,658 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 18:43:39,659 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34810, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 18:43:39,659 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 18:43:39,660 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39999] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,39999,1689187397821 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 18:43:39,660 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:39,661 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39999] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 148.251.75.209:34810 deadline: 1689187479660, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,39999,1689187397821 aborting 2023-07-12 18:43:39,660 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 18:43:39,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for f1f74138b310c7bba5d5da0cd13ec9ca: 2023-07-12 18:43:39,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:39,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 01c6221c5c54d75c1882a9c5fc3b040c, disabling compactions & flushes 2023-07-12 18:43:39,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:39,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:39,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. after waiting 0 ms 2023-07-12 18:43:39,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:39,663 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:39,663 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:39,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 01c6221c5c54d75c1882a9c5fc3b040c: 2023-07-12 18:43:39,664 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:39,764 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39999] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,39999,1689187397821 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 18:43:39,765 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39999] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 148.251.75.209:34810 deadline: 1689187479764, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,39999,1689187397821 aborting 2023-07-12 18:43:39,855 INFO [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,39999,1689187397821; all regions closed. 2023-07-12 18:43:39,871 DEBUG [RS:0;jenkins-hbase20:39999] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 18:43:39,871 INFO [RS:0;jenkins-hbase20:39999] regionserver.LeaseManager(133): Closed leases 2023-07-12 18:43:39,871 INFO [RS:0;jenkins-hbase20:39999] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 18:43:39,872 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 18:43:39,872 INFO [RS:0;jenkins-hbase20:39999] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:39999 2023-07-12 18:43:39,875 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 18:43:39,875 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:39,875 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 18:43:39,876 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,39999,1689187397821] 2023-07-12 18:43:39,876 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,39999,1689187397821; numProcessing=1 2023-07-12 18:43:39,877 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,39999,1689187397821 already deleted, retry=false 2023-07-12 18:43:39,877 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase20.apache.org,39999,1689187397821 on jenkins-hbase20.apache.org,33301,1689187396898 2023-07-12 18:43:39,877 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:41053-0x100827ea6a00008, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,39999,1689187397821 2023-07-12 18:43:39,877 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:41053-0x100827ea6a00008, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 18:43:39,878 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=17, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase20.apache.org,39999,1689187397821, splitWal=true, meta=true 2023-07-12 18:43:39,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:41053-0x100827ea6a00008, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:39,878 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=17 for jenkins-hbase20.apache.org,39999,1689187397821 (carryingMeta=true) jenkins-hbase20.apache.org,39999,1689187397821/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@2f2bbf60[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-12 18:43:39,880 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase20.apache.org,39999,1689187397821 znode expired, triggering replicatorRemoved event 2023-07-12 18:43:39,881 INFO [PEWorker-5] procedure.ServerCrashProcedure(161): Start pid=17, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,39999,1689187397821, splitWal=true, meta=true 2023-07-12 18:43:39,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:41053-0x100827ea6a00008, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:39,883 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,39999,1689187397821, splitWal=true, meta=true, isMeta: true 2023-07-12 18:43:39,887 DEBUG [PEWorker-5] master.MasterWalManager(318): Renamed region directory: hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting 2023-07-12 18:43:39,889 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase20.apache.org,39999,1689187397821 WAL count=1, meta=true 2023-07-12 18:43:39,890 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta}] 2023-07-12 18:43:39,892 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:39,893 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta, worker=jenkins-hbase20.apache.org,41053,1689187402445}] 2023-07-12 18:43:39,925 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 18:43:39,967 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:39999 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:39999 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 18:43:39,968 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:39999 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:39999 2023-07-12 18:43:39,976 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 18:43:39,976 INFO [RS:0;jenkins-hbase20:39999] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,39999,1689187397821; zookeeper connection closed. 2023-07-12 18:43:39,976 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:39999-0x100827ea6a00001, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 18:43:39,977 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5c625c39] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5c625c39 2023-07-12 18:43:40,047 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:40,047 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 18:43:40,049 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55872, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 18:43:40,049 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=41053] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=19 2023-07-12 18:43:40,064 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta, size=4.0 K (4130bytes) 2023-07-12 18:43:40,064 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta 2023-07-12 18:43:40,064 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta after 0ms 2023-07-12 18:43:40,071 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(311): Open hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta took 7ms 2023-07-12 18:43:40,077 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-12 18:43:40,079 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta so closing down 2023-07-12 18:43:40,080 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-12 18:43:40,085 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta.temp 2023-07-12 18:43:40,086 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-12 18:43:40,099 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) 2023-07-12 18:43:40,101 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta.temp to hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-12 18:43:40,101 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(423): Processed 13 edits across 1 Regions in 29 ms; skipped=1; WAL=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta, size=4.0 K, length=4130, corrupted=false, cancelled=false 2023-07-12 18:43:40,102 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta, journal: Splitting hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta, size=4.0 K (4130bytes) at 1689187420064Finishing writing output for hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta so closing down at 1689187420079 (+15 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta.temp at 1689187420085 (+6 ms)3 split writer threads finished at 1689187420086 (+1 ms)Closed recovered edits writer path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) at 1689187420099 (+13 ms)Rename recovered edits hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta.temp to hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 at 1689187420101 (+2 ms)Processed 13 edits across 1 Regions in 29 ms; skipped=1; WAL=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta, size=4.0 K, length=4130, corrupted=false, cancelled=false at 1689187420101 2023-07-12 18:43:40,102 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta 2023-07-12 18:43:40,102 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=19 2023-07-12 18:43:40,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster(3641): Remote procedure done, pid=19 2023-07-12 18:43:40,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta to hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/oldWALs 2023-07-12 18:43:40,111 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=18 2023-07-12 18:43:40,111 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta, worker=jenkins-hbase20.apache.org,41053,1689187402445 in 215 msec 2023-07-12 18:43:40,113 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:40,115 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-12 18:43:40,115 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase20.apache.org%2C39999%2C1689187397821.meta.1689187399342.meta, worker=jenkins-hbase20.apache.org,41053,1689187402445 in 223 msec 2023-07-12 18:43:40,118 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting dir is empty, no logs to split. 2023-07-12 18:43:40,118 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase20.apache.org,39999,1689187397821 WAL count=0, meta=true 2023-07-12 18:43:40,118 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase20.apache.org,39999,1689187397821 WAL splitting is done? wals=0, meta=true 2023-07-12 18:43:40,120 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 18:43:40,121 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 18:43:40,123 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 18:43:40,272 DEBUG [hconnection-0xdab2b0a-metaLookup-shared--pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:39999 this server is in the failed servers list 2023-07-12 18:43:40,273 DEBUG [jenkins-hbase20:33301] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 18:43:40,273 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,41053,1689187402445, state=OPENING 2023-07-12 18:43:40,274 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 18:43:40,274 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 18:43:40,275 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,41053,1689187402445}] 2023-07-12 18:43:40,434 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 18:43:40,434 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 18:43:40,438 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C41053%2C1689187402445.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,41053,1689187402445, archiveDir=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/oldWALs, maxLogs=32 2023-07-12 18:43:40,454 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45039,DS-3d7629ec-da28-4d9a-b55e-6763e4437bb1,DISK] 2023-07-12 18:43:40,454 DEBUG [RS-EventLoopGroup-6-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42053,DS-4ad92ce8-6d56-445f-9b36-b8468e44fc58,DISK] 2023-07-12 18:43:40,457 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,41053,1689187402445/jenkins-hbase20.apache.org%2C41053%2C1689187402445.meta.1689187420439.meta 2023-07-12 18:43:40,458 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45039,DS-3d7629ec-da28-4d9a-b55e-6763e4437bb1,DISK], DatanodeInfoWithStorage[127.0.0.1:42053,DS-4ad92ce8-6d56-445f-9b36-b8468e44fc58,DISK]] 2023-07-12 18:43:40,458 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 18:43:40,459 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 18:43:40,459 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 18:43:40,460 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 18:43:40,460 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 18:43:40,461 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 18:43:40,461 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 18:43:40,461 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 18:43:40,465 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 18:43:40,467 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/info 2023-07-12 18:43:40,467 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/info 2023-07-12 18:43:40,468 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 18:43:40,468 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 18:43:40,469 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 18:43:40,471 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/rep_barrier 2023-07-12 18:43:40,471 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/rep_barrier 2023-07-12 18:43:40,471 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 18:43:40,472 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 18:43:40,472 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 18:43:40,474 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/table 2023-07-12 18:43:40,474 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/table 2023-07-12 18:43:40,475 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 18:43:40,476 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 18:43:40,477 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740 2023-07-12 18:43:40,480 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740 2023-07-12 18:43:40,481 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-12 18:43:40,487 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5464): Applied 32, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=15, path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-12 18:43:40,487 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=3.16 KB heapSize=6.27 KB 2023-07-12 18:43:40,504 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.00 KB at sequenceid=15 (bloomFilter=false), to=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/.tmp/info/bec4e2ad5605432e9e1eeb4934027e10 2023-07-12 18:43:40,531 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=15 (bloomFilter=false), to=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/.tmp/table/21b9b1221d4246d39ae3a061be2e0284 2023-07-12 18:43:40,541 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/.tmp/info/bec4e2ad5605432e9e1eeb4934027e10 as hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/info/bec4e2ad5605432e9e1eeb4934027e10 2023-07-12 18:43:40,552 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/info/bec4e2ad5605432e9e1eeb4934027e10, entries=28, sequenceid=15, filesize=7.7 K 2023-07-12 18:43:40,554 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/.tmp/table/21b9b1221d4246d39ae3a061be2e0284 as hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/table/21b9b1221d4246d39ae3a061be2e0284 2023-07-12 18:43:40,562 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/table/21b9b1221d4246d39ae3a061be2e0284, entries=4, sequenceid=15, filesize=4.7 K 2023-07-12 18:43:40,562 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.16 KB/3234, heapSize ~5.98 KB/6128, currentSize=0 B/0 for 1588230740 in 75ms, sequenceid=15, compaction requested=false; wal=null 2023-07-12 18:43:40,564 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-12 18:43:40,568 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 18:43:40,569 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 18:43:40,572 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=1 2023-07-12 18:43:40,573 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73568768, jitterRate=0.09626007080078125}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 18:43:40,574 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 18:43:40,575 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=21, masterSystemTime=1689187420428 2023-07-12 18:43:40,576 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 18:43:40,577 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 18:43:40,578 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,41053,1689187402445, state=OPEN 2023-07-12 18:43:40,579 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 18:43:40,579 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 18:43:40,581 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-12 18:43:40,581 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,41053,1689187402445 in 305 msec 2023-07-12 18:43:40,584 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=17 2023-07-12 18:43:40,584 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase20.apache.org,39999,1689187397821 had 3 regions 2023-07-12 18:43:40,584 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 461 msec 2023-07-12 18:43:40,585 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,39999,1689187397821, splitWal=true, meta=true, isMeta: false 2023-07-12 18:43:40,588 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase20.apache.org,39999,1689187397821 WAL count=1, meta=false 2023-07-12 18:43:40,588 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100}] 2023-07-12 18:43:40,591 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:40,592 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100, worker=jenkins-hbase20.apache.org,41053,1689187402445}] 2023-07-12 18:43:40,746 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=41053] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=23 2023-07-12 18:43:40,763 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100, size=901 (901bytes) 2023-07-12 18:43:40,763 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100 2023-07-12 18:43:40,764 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100 after 1ms 2023-07-12 18:43:40,769 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(311): Open hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100 took 6ms 2023-07-12 18:43:40,773 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(346): Last flushed sequenceid for 01c6221c5c54d75c1882a9c5fc3b040c: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-12 18:43:40,775 DEBUG [hconnection-0xdab2b0a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 18:43:40,776 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(346): Last flushed sequenceid for f1f74138b310c7bba5d5da0cd13ec9ca: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-12 18:43:40,776 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100 so closing down 2023-07-12 18:43:40,777 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-12 18:43:40,778 INFO [RS-EventLoopGroup-6-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55886, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 18:43:40,780 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100.temp 2023-07-12 18:43:40,780 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-12 18:43:40,790 DEBUG [hconnection-0xdab2b0a-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:39999 this server is in the failed servers list 2023-07-12 18:43:40,792 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-12 18:43:40,794 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100.temp to hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/recovered.edits/0000000000000000005 2023-07-12 18:43:40,794 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(423): Processed 4 edits across 1 Regions in 24 ms; skipped=2; WAL=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100, size=901, length=901, corrupted=false, cancelled=false 2023-07-12 18:43:40,794 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100, journal: Splitting hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100, size=901 (901bytes) at 1689187420763Finishing writing output for hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100 so closing down at 1689187420777 (+14 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100.temp at 1689187420780 (+3 ms)3 split writer threads finished at 1689187420780Closed recovered edits writer path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689187420792 (+12 ms)Rename recovered edits hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100.temp to hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/recovered.edits/0000000000000000005 at 1689187420794 (+2 ms)Processed 4 edits across 1 Regions in 24 ms; skipped=2; WAL=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100, size=901, length=901, corrupted=false, cancelled=false at 1689187420794 2023-07-12 18:43:40,794 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100 2023-07-12 18:43:40,794 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=23 2023-07-12 18:43:40,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster(3641): Remote procedure done, pid=23 2023-07-12 18:43:40,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting/jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100 to hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/oldWALs 2023-07-12 18:43:40,802 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-12 18:43:40,802 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100, worker=jenkins-hbase20.apache.org,41053,1689187402445 in 207 msec 2023-07-12 18:43:40,804 DEBUG [PEWorker-2] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:40,806 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=17 2023-07-12 18:43:40,806 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase20.apache.org%2C39999%2C1689187397821.1689187399100, worker=jenkins-hbase20.apache.org,41053,1689187402445 in 216 msec 2023-07-12 18:43:40,809 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/WALs/jenkins-hbase20.apache.org,39999,1689187397821-splitting dir is empty, no logs to split. 2023-07-12 18:43:40,809 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase20.apache.org,39999,1689187397821 WAL count=0, meta=false 2023-07-12 18:43:40,809 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase20.apache.org,39999,1689187397821 WAL splitting is done? wals=0, meta=false 2023-07-12 18:43:40,811 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=01c6221c5c54d75c1882a9c5fc3b040c, ASSIGN}, {pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f1f74138b310c7bba5d5da0cd13ec9ca, ASSIGN}] 2023-07-12 18:43:40,813 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=01c6221c5c54d75c1882a9c5fc3b040c, ASSIGN 2023-07-12 18:43:40,813 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f1f74138b310c7bba5d5da0cd13ec9ca, ASSIGN 2023-07-12 18:43:40,815 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=f1f74138b310c7bba5d5da0cd13ec9ca, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 18:43:40,815 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=01c6221c5c54d75c1882a9c5fc3b040c, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-12 18:43:40,815 DEBUG [jenkins-hbase20:33301] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 18:43:40,817 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=f1f74138b310c7bba5d5da0cd13ec9ca, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:40,817 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=01c6221c5c54d75c1882a9c5fc3b040c, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:40,817 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689187420817"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689187420817"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689187420817"}]},"ts":"1689187420817"} 2023-07-12 18:43:40,817 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689187420817"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689187420817"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689187420817"}]},"ts":"1689187420817"} 2023-07-12 18:43:40,818 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:39999 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:39999 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 18:43:40,821 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:39999 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:39999 2023-07-12 18:43:40,901 DEBUG [hconnection-0xdab2b0a-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:39999 this server is in the failed servers list 2023-07-12 18:43:40,923 DEBUG [PEWorker-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 18:43:40,924 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55902, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 18:43:40,926 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; OpenRegionProcedure f1f74138b310c7bba5d5da0cd13ec9ca, server=jenkins-hbase20.apache.org,41053,1689187402445}] 2023-07-12 18:43:40,928 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=24, state=RUNNABLE; OpenRegionProcedure 01c6221c5c54d75c1882a9c5fc3b040c, server=jenkins-hbase20.apache.org,41053,1689187402445}] 2023-07-12 18:43:41,088 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:41,088 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f1f74138b310c7bba5d5da0cd13ec9ca, NAME => 't,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.', STARTKEY => '', ENDKEY => ''} 2023-07-12 18:43:41,088 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:41,088 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 18:43:41,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:41,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:41,091 INFO [StoreOpener-f1f74138b310c7bba5d5da0cd13ec9ca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:41,093 DEBUG [StoreOpener-f1f74138b310c7bba5d5da0cd13ec9ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/f 2023-07-12 18:43:41,093 DEBUG [StoreOpener-f1f74138b310c7bba5d5da0cd13ec9ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/f 2023-07-12 18:43:41,093 INFO [StoreOpener-f1f74138b310c7bba5d5da0cd13ec9ca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f1f74138b310c7bba5d5da0cd13ec9ca columnFamilyName f 2023-07-12 18:43:41,094 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:41,094 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 01c6221c5c54d75c1882a9c5fc3b040c, NAME => 'hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c.', STARTKEY => '', ENDKEY => ''} 2023-07-12 18:43:41,094 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:41,094 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 18:43:41,094 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:41,094 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:41,096 INFO [StoreOpener-01c6221c5c54d75c1882a9c5fc3b040c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:41,099 DEBUG [StoreOpener-01c6221c5c54d75c1882a9c5fc3b040c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/info 2023-07-12 18:43:41,099 DEBUG [StoreOpener-01c6221c5c54d75c1882a9c5fc3b040c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/info 2023-07-12 18:43:41,100 INFO [StoreOpener-01c6221c5c54d75c1882a9c5fc3b040c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 01c6221c5c54d75c1882a9c5fc3b040c columnFamilyName info 2023-07-12 18:43:41,101 INFO [StoreOpener-01c6221c5c54d75c1882a9c5fc3b040c-1] regionserver.HStore(310): Store=01c6221c5c54d75c1882a9c5fc3b040c/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 18:43:41,103 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:41,107 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:41,108 DEBUG [StoreOpener-f1f74138b310c7bba5d5da0cd13ec9ca-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/f/7fd593e96e85408da538ffb739043cbe 2023-07-12 18:43:41,108 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/recovered.edits/0000000000000000005 2023-07-12 18:43:41,108 INFO [StoreOpener-f1f74138b310c7bba5d5da0cd13ec9ca-1] regionserver.HStore(310): Store=f1f74138b310c7bba5d5da0cd13ec9ca/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 18:43:41,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:41,111 DEBUG [hconnection-0xdab2b0a-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:39999 this server is in the failed servers list 2023-07-12 18:43:41,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:41,113 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/recovered.edits/0000000000000000005 2023-07-12 18:43:41,113 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 01c6221c5c54d75c1882a9c5fc3b040c 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-12 18:43:41,115 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:41,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-12 18:43:41,118 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened f1f74138b310c7bba5d5da0cd13ec9ca; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75076723, jitterRate=0.11873035132884979}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 18:43:41,118 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for f1f74138b310c7bba5d5da0cd13ec9ca: 2023-07-12 18:43:41,119 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca., pid=26, masterSystemTime=1689187421079 2023-07-12 18:43:41,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:41,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:41,122 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=f1f74138b310c7bba5d5da0cd13ec9ca, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:41,123 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689187421122"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689187421122"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689187421122"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689187421122"}]},"ts":"1689187421122"} 2023-07-12 18:43:41,139 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-07-12 18:43:41,139 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; OpenRegionProcedure f1f74138b310c7bba5d5da0cd13ec9ca, server=jenkins-hbase20.apache.org,41053,1689187402445 in 199 msec 2023-07-12 18:43:41,142 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=t, region=f1f74138b310c7bba5d5da0cd13ec9ca, ASSIGN in 328 msec 2023-07-12 18:43:41,143 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/.tmp/info/bff64c4469f14338bf549be5b306a0e5 2023-07-12 18:43:41,152 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/.tmp/info/bff64c4469f14338bf549be5b306a0e5 as hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/info/bff64c4469f14338bf549be5b306a0e5 2023-07-12 18:43:41,161 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/info/bff64c4469f14338bf549be5b306a0e5, entries=2, sequenceid=5, filesize=4.8 K 2023-07-12 18:43:41,161 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 01c6221c5c54d75c1882a9c5fc3b040c in 48ms, sequenceid=5, compaction requested=false; wal=null 2023-07-12 18:43:41,163 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/recovered.edits/0000000000000000005 2023-07-12 18:43:41,166 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:41,169 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-12 18:43:41,170 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 01c6221c5c54d75c1882a9c5fc3b040c; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67165921, jitterRate=8.502155542373657E-4}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 18:43:41,171 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 01c6221c5c54d75c1882a9c5fc3b040c: 2023-07-12 18:43:41,172 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c., pid=27, masterSystemTime=1689187421079 2023-07-12 18:43:41,174 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:41,174 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:41,174 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=01c6221c5c54d75c1882a9c5fc3b040c, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:41,175 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689187421174"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689187421174"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689187421174"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689187421174"}]},"ts":"1689187421174"} 2023-07-12 18:43:41,180 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=24 2023-07-12 18:43:41,180 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=24, state=SUCCESS; OpenRegionProcedure 01c6221c5c54d75c1882a9c5fc3b040c, server=jenkins-hbase20.apache.org,41053,1689187402445 in 250 msec 2023-07-12 18:43:41,183 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=17 2023-07-12 18:43:41,184 INFO [PEWorker-4] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase20.apache.org,39999,1689187397821 after splitting done 2023-07-12 18:43:41,184 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=01c6221c5c54d75c1882a9c5fc3b040c, ASSIGN in 369 msec 2023-07-12 18:43:41,184 DEBUG [PEWorker-4] master.DeadServer(114): Removed jenkins-hbase20.apache.org,39999,1689187397821 from processing; numProcessing=0 2023-07-12 18:43:41,186 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, state=SUCCESS; ServerCrashProcedure jenkins-hbase20.apache.org,39999,1689187397821, splitWal=true, meta=true in 1.3070 sec 2023-07-12 18:43:41,431 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x53953132 to 127.0.0.1:63743 2023-07-12 18:43:41,431 DEBUG [Listener at localhost.localdomain/46185] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 18:43:41,431 INFO [Listener at localhost.localdomain/46185] client.TestScannerTimeout(124): END ************ test2772 2023-07-12 18:43:41,446 INFO [Listener at localhost.localdomain/46185] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=275 (was 293), OpenFileDescriptor=580 (was 571) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=167 (was 167), ProcessCount=167 (was 167), AvailableMemoryMB=6633 (was 6655) 2023-07-12 18:43:41,447 INFO [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 18:43:41,447 INFO [Listener at localhost.localdomain/46185] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 18:43:41,447 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0af4a250 to 127.0.0.1:63743 2023-07-12 18:43:41,447 DEBUG [Listener at localhost.localdomain/46185] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 18:43:41,447 DEBUG [Listener at localhost.localdomain/46185] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 18:43:41,448 DEBUG [Listener at localhost.localdomain/46185] util.JVMClusterUtil(257): Found active master hash=1741655023, stopped=false 2023-07-12 18:43:41,448 INFO [Listener at localhost.localdomain/46185] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,33301,1689187396898 2023-07-12 18:43:41,451 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 18:43:41,451 INFO [Listener at localhost.localdomain/46185] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 18:43:41,451 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:41053-0x100827ea6a00008, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 18:43:41,451 DEBUG [Listener at localhost.localdomain/46185] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0007cd6a to 127.0.0.1:63743 2023-07-12 18:43:41,451 DEBUG [Listener at localhost.localdomain/46185] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 18:43:41,451 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 18:43:41,452 INFO [Listener at localhost.localdomain/46185] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,41053,1689187402445' ***** 2023-07-12 18:43:41,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41053-0x100827ea6a00008, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 18:43:41,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 18:43:41,452 INFO [Listener at localhost.localdomain/46185] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 18:43:41,452 INFO [RS:2;jenkins-hbase20:41053] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 18:43:41,452 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 18:43:41,452 INFO [RS:2;jenkins-hbase20:41053] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 18:43:41,453 INFO [RS:2;jenkins-hbase20:41053] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 18:43:41,453 INFO [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(3305): Received CLOSE for f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:41,453 INFO [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(3305): Received CLOSE for 01c6221c5c54d75c1882a9c5fc3b040c 2023-07-12 18:43:41,453 INFO [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:41,453 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing f1f74138b310c7bba5d5da0cd13ec9ca, disabling compactions & flushes 2023-07-12 18:43:41,453 DEBUG [RS:2;jenkins-hbase20:41053] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x308c5166 to 127.0.0.1:63743 2023-07-12 18:43:41,454 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:41,454 DEBUG [RS:2;jenkins-hbase20:41053] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 18:43:41,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:41,454 INFO [RS:2;jenkins-hbase20:41053] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 18:43:41,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. after waiting 0 ms 2023-07-12 18:43:41,454 INFO [RS:2;jenkins-hbase20:41053] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 18:43:41,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:41,454 INFO [RS:2;jenkins-hbase20:41053] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 18:43:41,454 INFO [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 18:43:41,454 INFO [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-12 18:43:41,454 DEBUG [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(1478): Online Regions={f1f74138b310c7bba5d5da0cd13ec9ca=t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca., 1588230740=hbase:meta,,1.1588230740, 01c6221c5c54d75c1882a9c5fc3b040c=hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c.} 2023-07-12 18:43:41,454 DEBUG [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(1504): Waiting on 01c6221c5c54d75c1882a9c5fc3b040c, 1588230740, f1f74138b310c7bba5d5da0cd13ec9ca 2023-07-12 18:43:41,456 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 18:43:41,456 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 18:43:41,457 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 18:43:41,457 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 18:43:41,457 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 18:43:41,457 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-12 18:43:41,462 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/default/t/f1f74138b310c7bba5d5da0cd13ec9ca/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-12 18:43:41,463 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:41,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for f1f74138b310c7bba5d5da0cd13ec9ca: 2023-07-12 18:43:41,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed t,,1689187400463.f1f74138b310c7bba5d5da0cd13ec9ca. 2023-07-12 18:43:41,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 01c6221c5c54d75c1882a9c5fc3b040c, disabling compactions & flushes 2023-07-12 18:43:41,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:41,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:41,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. after waiting 0 ms 2023-07-12 18:43:41,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:41,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/namespace/01c6221c5c54d75c1882a9c5fc3b040c/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=5 2023-07-12 18:43:41,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:41,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 01c6221c5c54d75c1882a9c5fc3b040c: 2023-07-12 18:43:41,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689187399621.01c6221c5c54d75c1882a9c5fc3b040c. 2023-07-12 18:43:41,490 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=22 (bloomFilter=false), to=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/.tmp/info/5b773533924147deb9d65543bf92a983 2023-07-12 18:43:41,497 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/.tmp/info/5b773533924147deb9d65543bf92a983 as hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/info/5b773533924147deb9d65543bf92a983 2023-07-12 18:43:41,504 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/info/5b773533924147deb9d65543bf92a983, entries=16, sequenceid=22, filesize=6.4 K 2023-07-12 18:43:41,506 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1796, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 49ms, sequenceid=22, compaction requested=false 2023-07-12 18:43:41,513 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/data/hbase/meta/1588230740/recovered.edits/25.seqid, newMaxSeqId=25, maxSeqId=15 2023-07-12 18:43:41,514 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 18:43:41,514 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 18:43:41,514 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 18:43:41,514 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 18:43:41,514 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 18:43:41,577 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-12 18:43:41,577 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-12 18:43:41,655 INFO [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,41053,1689187402445; all regions closed. 2023-07-12 18:43:41,664 DEBUG [RS:2;jenkins-hbase20:41053] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/oldWALs 2023-07-12 18:43:41,664 INFO [RS:2;jenkins-hbase20:41053] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C41053%2C1689187402445.meta:.meta(num 1689187420439) 2023-07-12 18:43:41,671 DEBUG [RS:2;jenkins-hbase20:41053] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/oldWALs 2023-07-12 18:43:41,671 INFO [RS:2;jenkins-hbase20:41053] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C41053%2C1689187402445:(num 1689187402644) 2023-07-12 18:43:41,671 DEBUG [RS:2;jenkins-hbase20:41053] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 18:43:41,671 INFO [RS:2;jenkins-hbase20:41053] regionserver.LeaseManager(133): Closed leases 2023-07-12 18:43:41,671 INFO [RS:2;jenkins-hbase20:41053] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 18:43:41,671 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 18:43:41,672 INFO [RS:2;jenkins-hbase20:41053] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:41053 2023-07-12 18:43:41,673 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:41053-0x100827ea6a00008, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,41053,1689187402445 2023-07-12 18:43:41,673 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 18:43:41,673 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:41053-0x100827ea6a00008, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 18:43:41,675 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,41053,1689187402445] 2023-07-12 18:43:41,675 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,41053,1689187402445; numProcessing=1 2023-07-12 18:43:41,675 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,41053,1689187402445 already deleted, retry=false 2023-07-12 18:43:41,675 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,41053,1689187402445 expired; onlineServers=0 2023-07-12 18:43:41,675 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,33301,1689187396898' ***** 2023-07-12 18:43:41,675 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-12 18:43:41,676 DEBUG [M:0;jenkins-hbase20:33301] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@26f281d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-12 18:43:41,676 INFO [M:0;jenkins-hbase20:33301] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,33301,1689187396898 2023-07-12 18:43:41,676 INFO [M:0;jenkins-hbase20:33301] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,33301,1689187396898; all regions closed. 2023-07-12 18:43:41,676 DEBUG [M:0;jenkins-hbase20:33301] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 18:43:41,676 DEBUG [M:0;jenkins-hbase20:33301] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 18:43:41,676 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 18:43:41,677 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689187398829] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689187398829,5,FailOnTimeoutGroup] 2023-07-12 18:43:41,677 DEBUG [M:0;jenkins-hbase20:33301] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 18:43:41,677 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689187398831] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689187398831,5,FailOnTimeoutGroup] 2023-07-12 18:43:41,677 INFO [M:0;jenkins-hbase20:33301] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 18:43:41,677 INFO [M:0;jenkins-hbase20:33301] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 18:43:41,677 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 18:43:41,677 INFO [M:0;jenkins-hbase20:33301] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-07-12 18:43:41,677 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 18:43:41,677 DEBUG [M:0;jenkins-hbase20:33301] master.HMaster(1512): Stopping service threads 2023-07-12 18:43:41,677 INFO [M:0;jenkins-hbase20:33301] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 18:43:41,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 18:43:41,677 ERROR [M:0;jenkins-hbase20:33301] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (1091787570) connection to localhost.localdomain/127.0.0.1:33493 from jenkins,5,PEWorkerGroup] 2023-07-12 18:43:41,678 INFO [M:0;jenkins-hbase20:33301] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 18:43:41,678 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 18:43:41,678 DEBUG [M:0;jenkins-hbase20:33301] zookeeper.ZKUtil(398): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 18:43:41,678 WARN [M:0;jenkins-hbase20:33301] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 18:43:41,678 INFO [M:0;jenkins-hbase20:33301] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 18:43:41,678 INFO [M:0;jenkins-hbase20:33301] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-12 18:43:41,679 DEBUG [M:0;jenkins-hbase20:33301] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 18:43:41,679 INFO [M:0;jenkins-hbase20:33301] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 18:43:41,679 DEBUG [M:0;jenkins-hbase20:33301] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 18:43:41,679 DEBUG [M:0;jenkins-hbase20:33301] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 18:43:41,679 DEBUG [M:0;jenkins-hbase20:33301] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 18:43:41,679 INFO [M:0;jenkins-hbase20:33301] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=91.64 KB heapSize=110.36 KB 2023-07-12 18:43:41,694 INFO [M:0;jenkins-hbase20:33301] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=91.64 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/83367c29cd0c4c90a776519393a83cd9 2023-07-12 18:43:41,700 DEBUG [M:0;jenkins-hbase20:33301] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/83367c29cd0c4c90a776519393a83cd9 as hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/83367c29cd0c4c90a776519393a83cd9 2023-07-12 18:43:41,707 INFO [M:0;jenkins-hbase20:33301] regionserver.HStore(1080): Added hdfs://localhost.localdomain:33493/user/jenkins/test-data/a70c9ecf-5f9d-7fa0-d225-c9946f2c7c1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/83367c29cd0c4c90a776519393a83cd9, entries=27, sequenceid=212, filesize=8.2 K 2023-07-12 18:43:41,708 INFO [M:0;jenkins-hbase20:33301] regionserver.HRegion(2948): Finished flush of dataSize ~91.64 KB/93836, heapSize ~110.34 KB/112992, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 29ms, sequenceid=212, compaction requested=false 2023-07-12 18:43:41,710 INFO [M:0;jenkins-hbase20:33301] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 18:43:41,710 DEBUG [M:0;jenkins-hbase20:33301] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 18:43:41,713 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 18:43:41,713 INFO [M:0;jenkins-hbase20:33301] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 18:43:41,714 INFO [M:0;jenkins-hbase20:33301] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:33301 2023-07-12 18:43:41,716 DEBUG [M:0;jenkins-hbase20:33301] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,33301,1689187396898 already deleted, retry=false 2023-07-12 18:43:41,937 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 18:43:41,937 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): master:33301-0x100827ea6a00000, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 18:43:41,937 INFO [M:0;jenkins-hbase20:33301] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,33301,1689187396898; zookeeper connection closed. 2023-07-12 18:43:42,037 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:41053-0x100827ea6a00008, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 18:43:42,037 DEBUG [Listener at localhost.localdomain/46185-EventThread] zookeeper.ZKWatcher(600): regionserver:41053-0x100827ea6a00008, quorum=127.0.0.1:63743, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 18:43:42,037 INFO [RS:2;jenkins-hbase20:41053] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,41053,1689187402445; zookeeper connection closed. 2023-07-12 18:43:42,038 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2ed46294] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2ed46294 2023-07-12 18:43:42,039 INFO [Listener at localhost.localdomain/46185] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-12 18:43:42,039 WARN [Listener at localhost.localdomain/46185] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 18:43:42,046 INFO [Listener at localhost.localdomain/46185] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 18:43:42,157 WARN [BP-775451423-148.251.75.209-1689187394261 heartbeating to localhost.localdomain/127.0.0.1:33493] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 18:43:42,158 WARN [BP-775451423-148.251.75.209-1689187394261 heartbeating to localhost.localdomain/127.0.0.1:33493] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-775451423-148.251.75.209-1689187394261 (Datanode Uuid 019ccb5f-98d9-478b-9e9b-908bbab7e733) service to localhost.localdomain/127.0.0.1:33493 2023-07-12 18:43:42,159 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/cluster_55ffc31b-bad0-f043-ae3b-8971827a1352/dfs/data/data3/current/BP-775451423-148.251.75.209-1689187394261] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 18:43:42,159 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/cluster_55ffc31b-bad0-f043-ae3b-8971827a1352/dfs/data/data4/current/BP-775451423-148.251.75.209-1689187394261] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 18:43:42,160 WARN [Listener at localhost.localdomain/46185] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 18:43:42,162 INFO [Listener at localhost.localdomain/46185] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 18:43:42,270 WARN [BP-775451423-148.251.75.209-1689187394261 heartbeating to localhost.localdomain/127.0.0.1:33493] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 18:43:42,270 WARN [BP-775451423-148.251.75.209-1689187394261 heartbeating to localhost.localdomain/127.0.0.1:33493] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-775451423-148.251.75.209-1689187394261 (Datanode Uuid bc0640a0-7a83-4df7-8de8-a7ab5411d425) service to localhost.localdomain/127.0.0.1:33493 2023-07-12 18:43:42,272 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/cluster_55ffc31b-bad0-f043-ae3b-8971827a1352/dfs/data/data1/current/BP-775451423-148.251.75.209-1689187394261] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 18:43:42,273 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/93382b8e-42eb-7694-0f3c-c7ee6b7ac768/cluster_55ffc31b-bad0-f043-ae3b-8971827a1352/dfs/data/data2/current/BP-775451423-148.251.75.209-1689187394261] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 18:43:42,306 INFO [Listener at localhost.localdomain/46185] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-07-12 18:43:42,421 INFO [Listener at localhost.localdomain/46185] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 18:43:42,457 INFO [Listener at localhost.localdomain/46185] hbase.HBaseTestingUtility(1293): Minicluster is down