2023-07-15 11:05:35,023 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-15 11:05:35,265 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c 2023-07-15 11:05:35,279 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-15 11:05:35,280 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/cluster_502b0317-3284-cd7f-c819-f42948f0d6d4, deleteOnExit=true 2023-07-15 11:05:35,280 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-15 11:05:35,281 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/test.cache.data in system properties and HBase conf 2023-07-15 11:05:35,281 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/hadoop.tmp.dir in system properties and HBase conf 2023-07-15 11:05:35,282 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/hadoop.log.dir in system properties and HBase conf 2023-07-15 11:05:35,283 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-15 11:05:35,283 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-15 11:05:35,283 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-15 11:05:35,400 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-15 11:05:35,786 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-15 11:05:35,790 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-15 11:05:35,790 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-15 11:05:35,791 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-15 11:05:35,791 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-15 11:05:35,791 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-15 11:05:35,791 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-15 11:05:35,792 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-15 11:05:35,792 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-15 11:05:35,792 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-15 11:05:35,793 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/nfs.dump.dir in system properties and HBase conf 2023-07-15 11:05:35,793 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/java.io.tmpdir in system properties and HBase conf 2023-07-15 11:05:35,793 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-15 11:05:35,793 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-15 11:05:35,794 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-15 11:05:36,249 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-15 11:05:36,254 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-15 11:05:36,532 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-15 11:05:36,692 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-15 11:05:36,706 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-15 11:05:36,740 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-15 11:05:36,771 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/java.io.tmpdir/Jetty_localhost_40009_hdfs____.bdplax/webapp 2023-07-15 11:05:36,927 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40009 2023-07-15 11:05:36,937 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-15 11:05:36,938 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-15 11:05:37,359 WARN [Listener at localhost/33777] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-15 11:05:37,454 WARN [Listener at localhost/33777] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-15 11:05:37,472 WARN [Listener at localhost/33777] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-15 11:05:37,479 INFO [Listener at localhost/33777] log.Slf4jLog(67): jetty-6.1.26 2023-07-15 11:05:37,483 INFO [Listener at localhost/33777] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/java.io.tmpdir/Jetty_localhost_40305_datanode____.7cu4d9/webapp 2023-07-15 11:05:37,579 INFO [Listener at localhost/33777] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40305 2023-07-15 11:05:37,877 WARN [Listener at localhost/45121] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-15 11:05:37,890 WARN [Listener at localhost/45121] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-15 11:05:37,894 WARN [Listener at localhost/45121] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-15 11:05:37,896 INFO [Listener at localhost/45121] log.Slf4jLog(67): jetty-6.1.26 2023-07-15 11:05:37,901 INFO [Listener at localhost/45121] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/java.io.tmpdir/Jetty_localhost_42831_datanode____npo2m7/webapp 2023-07-15 11:05:37,998 INFO [Listener at localhost/45121] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42831 2023-07-15 11:05:38,009 WARN [Listener at localhost/42421] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-15 11:05:38,337 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd728f555279794a4: Processing first storage report for DS-e17b79f3-7f48-44a0-a435-8c634aa6dce9 from datanode 0f02e0a7-cac3-493c-8997-9fd577754c46 2023-07-15 11:05:38,339 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd728f555279794a4: from storage DS-e17b79f3-7f48-44a0-a435-8c634aa6dce9 node DatanodeRegistration(127.0.0.1:34387, datanodeUuid=0f02e0a7-cac3-493c-8997-9fd577754c46, infoPort=44903, infoSecurePort=0, ipcPort=45121, storageInfo=lv=-57;cid=testClusterID;nsid=1422620709;c=1689419136325), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-15 11:05:38,339 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x1c5424ea0ff25b58: Processing first storage report for DS-213137f8-2c0c-4882-b473-e4f5e74cacf4 from datanode cda3437d-c0d2-4e8a-a7b3-d735ed597d37 2023-07-15 11:05:38,340 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1c5424ea0ff25b58: from storage DS-213137f8-2c0c-4882-b473-e4f5e74cacf4 node DatanodeRegistration(127.0.0.1:32961, datanodeUuid=cda3437d-c0d2-4e8a-a7b3-d735ed597d37, infoPort=39307, infoSecurePort=0, ipcPort=42421, storageInfo=lv=-57;cid=testClusterID;nsid=1422620709;c=1689419136325), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 11:05:38,340 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd728f555279794a4: Processing first storage report for DS-d89b92bd-99b6-46bd-8c13-954eeb5c9d41 from datanode 0f02e0a7-cac3-493c-8997-9fd577754c46 2023-07-15 11:05:38,340 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd728f555279794a4: from storage DS-d89b92bd-99b6-46bd-8c13-954eeb5c9d41 node DatanodeRegistration(127.0.0.1:34387, datanodeUuid=0f02e0a7-cac3-493c-8997-9fd577754c46, infoPort=44903, infoSecurePort=0, ipcPort=45121, storageInfo=lv=-57;cid=testClusterID;nsid=1422620709;c=1689419136325), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 11:05:38,340 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x1c5424ea0ff25b58: Processing first storage report for DS-59c58c4e-3fee-452b-b6e7-b94d2aa8192a from datanode cda3437d-c0d2-4e8a-a7b3-d735ed597d37 2023-07-15 11:05:38,340 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1c5424ea0ff25b58: from storage DS-59c58c4e-3fee-452b-b6e7-b94d2aa8192a node DatanodeRegistration(127.0.0.1:32961, datanodeUuid=cda3437d-c0d2-4e8a-a7b3-d735ed597d37, infoPort=39307, infoSecurePort=0, ipcPort=42421, storageInfo=lv=-57;cid=testClusterID;nsid=1422620709;c=1689419136325), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-15 11:05:38,413 DEBUG [Listener at localhost/42421] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c 2023-07-15 11:05:38,478 INFO [Listener at localhost/42421] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/cluster_502b0317-3284-cd7f-c819-f42948f0d6d4/zookeeper_0, clientPort=63907, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/cluster_502b0317-3284-cd7f-c819-f42948f0d6d4/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/cluster_502b0317-3284-cd7f-c819-f42948f0d6d4/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-15 11:05:38,494 INFO [Listener at localhost/42421] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=63907 2023-07-15 11:05:38,502 INFO [Listener at localhost/42421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 11:05:38,504 INFO [Listener at localhost/42421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 11:05:39,165 INFO [Listener at localhost/42421] util.FSUtils(471): Created version file at hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be with version=8 2023-07-15 11:05:39,166 INFO [Listener at localhost/42421] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/hbase-staging 2023-07-15 11:05:39,470 INFO [Listener at localhost/42421] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-15 11:05:39,915 INFO [Listener at localhost/42421] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 11:05:39,946 INFO [Listener at localhost/42421] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 11:05:39,947 INFO [Listener at localhost/42421] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 11:05:39,947 INFO [Listener at localhost/42421] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 11:05:39,947 INFO [Listener at localhost/42421] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 11:05:39,947 INFO [Listener at localhost/42421] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 11:05:40,086 INFO [Listener at localhost/42421] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 11:05:40,158 DEBUG [Listener at localhost/42421] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-15 11:05:40,251 INFO [Listener at localhost/42421] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38773 2023-07-15 11:05:40,261 INFO [Listener at localhost/42421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 11:05:40,263 INFO [Listener at localhost/42421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 11:05:40,288 INFO [Listener at localhost/42421] zookeeper.RecoverableZooKeeper(93): Process identifier=master:38773 connecting to ZooKeeper ensemble=127.0.0.1:63907 2023-07-15 11:05:40,330 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:387730x0, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 11:05:40,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:38773-0x10168a8c89e0000 connected 2023-07-15 11:05:40,357 DEBUG [Listener at localhost/42421] zookeeper.ZKUtil(164): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 11:05:40,358 DEBUG [Listener at localhost/42421] zookeeper.ZKUtil(164): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 11:05:40,362 DEBUG [Listener at localhost/42421] zookeeper.ZKUtil(164): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 11:05:40,370 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38773 2023-07-15 11:05:40,371 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38773 2023-07-15 11:05:40,372 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38773 2023-07-15 11:05:40,373 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38773 2023-07-15 11:05:40,374 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38773 2023-07-15 11:05:40,380 INFO [Listener at localhost/42421] master.HMaster(444): hbase.rootdir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be, hbase.cluster.distributed=false 2023-07-15 11:05:40,447 INFO [Listener at localhost/42421] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 11:05:40,447 INFO [Listener at localhost/42421] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 11:05:40,448 INFO [Listener at localhost/42421] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 11:05:40,448 INFO [Listener at localhost/42421] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 11:05:40,448 INFO [Listener at localhost/42421] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 11:05:40,448 INFO [Listener at localhost/42421] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 11:05:40,453 INFO [Listener at localhost/42421] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 11:05:40,455 INFO [Listener at localhost/42421] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34427 2023-07-15 11:05:40,458 INFO [Listener at localhost/42421] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 11:05:40,464 DEBUG [Listener at localhost/42421] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 11:05:40,465 INFO [Listener at localhost/42421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 11:05:40,467 INFO [Listener at localhost/42421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 11:05:40,469 INFO [Listener at localhost/42421] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34427 connecting to ZooKeeper ensemble=127.0.0.1:63907 2023-07-15 11:05:40,473 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:344270x0, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 11:05:40,474 DEBUG [Listener at localhost/42421] zookeeper.ZKUtil(164): regionserver:344270x0, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 11:05:40,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:34427-0x10168a8c89e0001 connected 2023-07-15 11:05:40,476 DEBUG [Listener at localhost/42421] zookeeper.ZKUtil(164): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 11:05:40,477 DEBUG [Listener at localhost/42421] zookeeper.ZKUtil(164): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 11:05:40,477 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34427 2023-07-15 11:05:40,478 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34427 2023-07-15 11:05:40,478 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34427 2023-07-15 11:05:40,479 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34427 2023-07-15 11:05:40,479 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34427 2023-07-15 11:05:40,492 INFO [Listener at localhost/42421] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 11:05:40,492 INFO [Listener at localhost/42421] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 11:05:40,492 INFO [Listener at localhost/42421] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 11:05:40,493 INFO [Listener at localhost/42421] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 11:05:40,493 INFO [Listener at localhost/42421] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 11:05:40,493 INFO [Listener at localhost/42421] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 11:05:40,493 INFO [Listener at localhost/42421] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 11:05:40,494 INFO [Listener at localhost/42421] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41817 2023-07-15 11:05:40,495 INFO [Listener at localhost/42421] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 11:05:40,496 DEBUG [Listener at localhost/42421] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 11:05:40,496 INFO [Listener at localhost/42421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 11:05:40,498 INFO [Listener at localhost/42421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 11:05:40,500 INFO [Listener at localhost/42421] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41817 connecting to ZooKeeper ensemble=127.0.0.1:63907 2023-07-15 11:05:40,504 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:418170x0, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 11:05:40,505 DEBUG [Listener at localhost/42421] zookeeper.ZKUtil(164): regionserver:418170x0, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 11:05:40,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:41817-0x10168a8c89e0002 connected 2023-07-15 11:05:40,506 DEBUG [Listener at localhost/42421] zookeeper.ZKUtil(164): regionserver:41817-0x10168a8c89e0002, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 11:05:40,506 DEBUG [Listener at localhost/42421] zookeeper.ZKUtil(164): regionserver:41817-0x10168a8c89e0002, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 11:05:40,509 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41817 2023-07-15 11:05:40,509 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41817 2023-07-15 11:05:40,510 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41817 2023-07-15 11:05:40,510 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41817 2023-07-15 11:05:40,510 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41817 2023-07-15 11:05:40,512 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,38773,1689419139308 2023-07-15 11:05:40,520 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-15 11:05:40,521 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,38773,1689419139308 2023-07-15 11:05:40,540 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-15 11:05:40,540 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-15 11:05:40,540 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 11:05:40,540 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:41817-0x10168a8c89e0002, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-15 11:05:40,541 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 11:05:40,543 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,38773,1689419139308 from backup master directory 2023-07-15 11:05:40,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 11:05:40,546 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,38773,1689419139308 2023-07-15 11:05:40,546 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-15 11:05:40,547 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 11:05:40,547 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,38773,1689419139308 2023-07-15 11:05:40,550 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-15 11:05:40,551 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-15 11:05:40,631 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/hbase.id with ID: c245dd99-d642-43ea-a0ee-16a79478bf9b 2023-07-15 11:05:40,673 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 11:05:40,687 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 11:05:40,729 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x230e2d01 to 127.0.0.1:63907 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 11:05:40,761 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c066c8a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 11:05:40,786 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-15 11:05:40,788 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-15 11:05:40,806 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-15 11:05:40,807 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-15 11:05:40,809 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-15 11:05:40,813 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-15 11:05:40,814 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 11:05:40,849 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/MasterData/data/master/store-tmp 2023-07-15 11:05:40,881 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 11:05:40,882 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-15 11:05:40,882 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 11:05:40,882 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 11:05:40,882 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-15 11:05:40,882 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 11:05:40,882 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 11:05:40,882 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-15 11:05:40,884 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/MasterData/WALs/jenkins-hbase4.apache.org,38773,1689419139308 2023-07-15 11:05:40,905 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38773%2C1689419139308, suffix=, logDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/MasterData/WALs/jenkins-hbase4.apache.org,38773,1689419139308, archiveDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/MasterData/oldWALs, maxLogs=10 2023-07-15 11:05:40,960 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34387,DS-e17b79f3-7f48-44a0-a435-8c634aa6dce9,DISK] 2023-07-15 11:05:40,960 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32961,DS-213137f8-2c0c-4882-b473-e4f5e74cacf4,DISK] 2023-07-15 11:05:40,968 DEBUG [RS-EventLoopGroup-4-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 11:05:41,029 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/MasterData/WALs/jenkins-hbase4.apache.org,38773,1689419139308/jenkins-hbase4.apache.org%2C38773%2C1689419139308.1689419140915 2023-07-15 11:05:41,030 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32961,DS-213137f8-2c0c-4882-b473-e4f5e74cacf4,DISK], DatanodeInfoWithStorage[127.0.0.1:34387,DS-e17b79f3-7f48-44a0-a435-8c634aa6dce9,DISK]] 2023-07-15 11:05:41,031 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-15 11:05:41,031 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 11:05:41,034 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-15 11:05:41,035 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-15 11:05:41,092 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-15 11:05:41,098 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-15 11:05:41,121 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-15 11:05:41,133 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:05:41,139 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-15 11:05:41,141 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-15 11:05:41,155 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-15 11:05:41,159 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 11:05:41,161 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72421162, jitterRate=0.07915940880775452}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 11:05:41,161 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-15 11:05:41,162 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-15 11:05:41,182 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-15 11:05:41,182 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-15 11:05:41,184 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-15 11:05:41,186 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-15 11:05:41,217 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-07-15 11:05:41,217 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-15 11:05:41,243 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-15 11:05:41,248 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-15 11:05:41,273 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-15 11:05:41,276 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-15 11:05:41,277 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-15 11:05:41,282 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-15 11:05:41,286 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-15 11:05:41,289 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 11:05:41,290 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-15 11:05:41,291 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-15 11:05:41,302 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-15 11:05:41,306 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-15 11:05:41,306 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-15 11:05:41,306 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:41817-0x10168a8c89e0002, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-15 11:05:41,307 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 11:05:41,307 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,38773,1689419139308, sessionid=0x10168a8c89e0000, setting cluster-up flag (Was=false) 2023-07-15 11:05:41,321 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 11:05:41,327 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-15 11:05:41,329 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,38773,1689419139308 2023-07-15 11:05:41,334 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 11:05:41,340 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-15 11:05:41,341 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,38773,1689419139308 2023-07-15 11:05:41,344 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/.hbase-snapshot/.tmp 2023-07-15 11:05:41,414 INFO [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer(951): ClusterId : c245dd99-d642-43ea-a0ee-16a79478bf9b 2023-07-15 11:05:41,414 INFO [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(951): ClusterId : c245dd99-d642-43ea-a0ee-16a79478bf9b 2023-07-15 11:05:41,419 DEBUG [RS:0;jenkins-hbase4:34427] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 11:05:41,419 DEBUG [RS:1;jenkins-hbase4:41817] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 11:05:41,425 DEBUG [RS:0;jenkins-hbase4:34427] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 11:05:41,425 DEBUG [RS:1;jenkins-hbase4:41817] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 11:05:41,425 DEBUG [RS:0;jenkins-hbase4:34427] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 11:05:41,426 DEBUG [RS:1;jenkins-hbase4:41817] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 11:05:41,429 DEBUG [RS:0;jenkins-hbase4:34427] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 11:05:41,429 DEBUG [RS:1;jenkins-hbase4:41817] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 11:05:41,430 DEBUG [RS:0;jenkins-hbase4:34427] zookeeper.ReadOnlyZKClient(139): Connect 0x29c90dff to 127.0.0.1:63907 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 11:05:41,430 DEBUG [RS:1;jenkins-hbase4:41817] zookeeper.ReadOnlyZKClient(139): Connect 0x4ff98d89 to 127.0.0.1:63907 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 11:05:41,436 DEBUG [RS:0;jenkins-hbase4:34427] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@270ee6b6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 11:05:41,437 DEBUG [RS:0;jenkins-hbase4:34427] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2331668f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 11:05:41,437 DEBUG [RS:1;jenkins-hbase4:41817] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@413e6a06, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 11:05:41,438 DEBUG [RS:1;jenkins-hbase4:41817] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4501e6f9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 11:05:41,464 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-15 11:05:41,465 DEBUG [RS:1;jenkins-hbase4:41817] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:41817 2023-07-15 11:05:41,465 DEBUG [RS:0;jenkins-hbase4:34427] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:34427 2023-07-15 11:05:41,469 INFO [RS:0;jenkins-hbase4:34427] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 11:05:41,469 INFO [RS:1;jenkins-hbase4:41817] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 11:05:41,470 INFO [RS:0;jenkins-hbase4:34427] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 11:05:41,470 INFO [RS:1;jenkins-hbase4:41817] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 11:05:41,470 DEBUG [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 11:05:41,470 DEBUG [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 11:05:41,473 INFO [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,38773,1689419139308 with isa=jenkins-hbase4.apache.org/172.31.14.131:41817, startcode=1689419140491 2023-07-15 11:05:41,473 INFO [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,38773,1689419139308 with isa=jenkins-hbase4.apache.org/172.31.14.131:34427, startcode=1689419140446 2023-07-15 11:05:41,476 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 11:05:41,476 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 11:05:41,476 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 11:05:41,476 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-15 11:05:41,476 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-15 11:05:41,476 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,476 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 11:05:41,477 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,478 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689419171478 2023-07-15 11:05:41,480 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-15 11:05:41,484 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-15 11:05:41,485 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-15 11:05:41,490 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-15 11:05:41,491 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-15 11:05:41,492 DEBUG [RS:1;jenkins-hbase4:41817] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 11:05:41,492 DEBUG [RS:0;jenkins-hbase4:34427] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 11:05:41,498 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-15 11:05:41,498 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-15 11:05:41,499 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-15 11:05:41,499 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-15 11:05:41,499 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:41,501 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-15 11:05:41,502 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-15 11:05:41,503 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-15 11:05:41,505 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-15 11:05:41,506 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-15 11:05:41,507 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689419141507,5,FailOnTimeoutGroup] 2023-07-15 11:05:41,507 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689419141507,5,FailOnTimeoutGroup] 2023-07-15 11:05:41,507 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:41,507 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-15 11:05:41,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:41,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:41,533 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-15 11:05:41,534 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-15 11:05:41,534 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be 2023-07-15 11:05:41,553 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 11:05:41,556 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 11:05:41,559 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/info 2023-07-15 11:05:41,559 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 11:05:41,561 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:05:41,561 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 11:05:41,562 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50055, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 11:05:41,562 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45711, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 11:05:41,564 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/rep_barrier 2023-07-15 11:05:41,564 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 11:05:41,565 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:05:41,565 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 11:05:41,567 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/table 2023-07-15 11:05:41,568 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 11:05:41,569 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:05:41,570 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740 2023-07-15 11:05:41,571 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740 2023-07-15 11:05:41,573 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38773] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:41,575 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 11:05:41,578 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 11:05:41,582 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 11:05:41,583 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73882919, jitterRate=0.10094128549098969}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 11:05:41,583 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 11:05:41,583 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 11:05:41,583 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 11:05:41,583 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 11:05:41,583 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 11:05:41,583 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 11:05:41,584 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 11:05:41,584 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 11:05:41,586 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38773] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,41817,1689419140491 2023-07-15 11:05:41,591 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-15 11:05:41,591 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-15 11:05:41,593 DEBUG [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be 2023-07-15 11:05:41,593 DEBUG [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be 2023-07-15 11:05:41,593 DEBUG [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:33777 2023-07-15 11:05:41,593 DEBUG [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:33777 2023-07-15 11:05:41,593 DEBUG [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-15 11:05:41,593 DEBUG [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-15 11:05:41,599 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 11:05:41,600 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-15 11:05:41,603 DEBUG [RS:0;jenkins-hbase4:34427] zookeeper.ZKUtil(162): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:41,603 WARN [RS:0;jenkins-hbase4:34427] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 11:05:41,603 INFO [RS:0;jenkins-hbase4:34427] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 11:05:41,604 DEBUG [RS:1;jenkins-hbase4:41817] zookeeper.ZKUtil(162): regionserver:41817-0x10168a8c89e0002, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41817,1689419140491 2023-07-15 11:05:41,604 DEBUG [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(1948): logDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:41,604 WARN [RS:1;jenkins-hbase4:41817] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 11:05:41,604 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34427,1689419140446] 2023-07-15 11:05:41,604 INFO [RS:1;jenkins-hbase4:41817] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 11:05:41,605 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,41817,1689419140491] 2023-07-15 11:05:41,605 DEBUG [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer(1948): logDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491 2023-07-15 11:05:41,615 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-15 11:05:41,615 DEBUG [RS:1;jenkins-hbase4:41817] zookeeper.ZKUtil(162): regionserver:41817-0x10168a8c89e0002, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41817,1689419140491 2023-07-15 11:05:41,615 DEBUG [RS:0;jenkins-hbase4:34427] zookeeper.ZKUtil(162): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41817,1689419140491 2023-07-15 11:05:41,617 DEBUG [RS:0;jenkins-hbase4:34427] zookeeper.ZKUtil(162): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:41,617 DEBUG [RS:1;jenkins-hbase4:41817] zookeeper.ZKUtil(162): regionserver:41817-0x10168a8c89e0002, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:41,618 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-15 11:05:41,627 DEBUG [RS:0;jenkins-hbase4:34427] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 11:05:41,627 DEBUG [RS:1;jenkins-hbase4:41817] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 11:05:41,636 INFO [RS:0;jenkins-hbase4:34427] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 11:05:41,636 INFO [RS:1;jenkins-hbase4:41817] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 11:05:41,655 INFO [RS:1;jenkins-hbase4:41817] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 11:05:41,655 INFO [RS:0;jenkins-hbase4:34427] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 11:05:41,659 INFO [RS:1;jenkins-hbase4:41817] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 11:05:41,659 INFO [RS:0;jenkins-hbase4:34427] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 11:05:41,659 INFO [RS:1;jenkins-hbase4:41817] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:41,659 INFO [RS:0;jenkins-hbase4:34427] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:41,660 INFO [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 11:05:41,660 INFO [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 11:05:41,668 INFO [RS:1;jenkins-hbase4:41817] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:41,668 INFO [RS:0;jenkins-hbase4:34427] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:41,668 DEBUG [RS:1;jenkins-hbase4:41817] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,668 DEBUG [RS:0;jenkins-hbase4:34427] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,669 DEBUG [RS:1;jenkins-hbase4:41817] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,669 DEBUG [RS:0;jenkins-hbase4:34427] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,669 DEBUG [RS:1;jenkins-hbase4:41817] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,669 DEBUG [RS:0;jenkins-hbase4:34427] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,669 DEBUG [RS:1;jenkins-hbase4:41817] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,669 DEBUG [RS:0;jenkins-hbase4:34427] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,669 DEBUG [RS:1;jenkins-hbase4:41817] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,669 DEBUG [RS:0;jenkins-hbase4:34427] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,669 DEBUG [RS:1;jenkins-hbase4:41817] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 11:05:41,669 DEBUG [RS:0;jenkins-hbase4:34427] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 11:05:41,669 DEBUG [RS:1;jenkins-hbase4:41817] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,669 DEBUG [RS:0;jenkins-hbase4:34427] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,669 DEBUG [RS:1;jenkins-hbase4:41817] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,669 DEBUG [RS:0;jenkins-hbase4:34427] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,670 DEBUG [RS:1;jenkins-hbase4:41817] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,670 DEBUG [RS:0;jenkins-hbase4:34427] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,670 DEBUG [RS:1;jenkins-hbase4:41817] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,670 DEBUG [RS:0;jenkins-hbase4:34427] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:41,671 INFO [RS:0;jenkins-hbase4:34427] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:41,671 INFO [RS:0;jenkins-hbase4:34427] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:41,671 INFO [RS:0;jenkins-hbase4:34427] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:41,673 INFO [RS:1;jenkins-hbase4:41817] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:41,673 INFO [RS:1;jenkins-hbase4:41817] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:41,674 INFO [RS:1;jenkins-hbase4:41817] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:41,690 INFO [RS:1;jenkins-hbase4:41817] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 11:05:41,690 INFO [RS:0;jenkins-hbase4:34427] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 11:05:41,692 INFO [RS:1;jenkins-hbase4:41817] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41817,1689419140491-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:41,692 INFO [RS:0;jenkins-hbase4:34427] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34427,1689419140446-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:41,708 INFO [RS:1;jenkins-hbase4:41817] regionserver.Replication(203): jenkins-hbase4.apache.org,41817,1689419140491 started 2023-07-15 11:05:41,708 INFO [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,41817,1689419140491, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:41817, sessionid=0x10168a8c89e0002 2023-07-15 11:05:41,708 DEBUG [RS:1;jenkins-hbase4:41817] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 11:05:41,708 DEBUG [RS:1;jenkins-hbase4:41817] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,41817,1689419140491 2023-07-15 11:05:41,708 DEBUG [RS:1;jenkins-hbase4:41817] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41817,1689419140491' 2023-07-15 11:05:41,708 DEBUG [RS:1;jenkins-hbase4:41817] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 11:05:41,709 DEBUG [RS:1;jenkins-hbase4:41817] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 11:05:41,709 INFO [RS:0;jenkins-hbase4:34427] regionserver.Replication(203): jenkins-hbase4.apache.org,34427,1689419140446 started 2023-07-15 11:05:41,709 INFO [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,34427,1689419140446, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34427, sessionid=0x10168a8c89e0001 2023-07-15 11:05:41,709 DEBUG [RS:0;jenkins-hbase4:34427] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 11:05:41,709 DEBUG [RS:0;jenkins-hbase4:34427] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:41,710 DEBUG [RS:0;jenkins-hbase4:34427] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34427,1689419140446' 2023-07-15 11:05:41,710 DEBUG [RS:0;jenkins-hbase4:34427] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 11:05:41,710 DEBUG [RS:1;jenkins-hbase4:41817] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 11:05:41,710 DEBUG [RS:1;jenkins-hbase4:41817] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 11:05:41,711 DEBUG [RS:1;jenkins-hbase4:41817] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,41817,1689419140491 2023-07-15 11:05:41,711 DEBUG [RS:1;jenkins-hbase4:41817] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41817,1689419140491' 2023-07-15 11:05:41,711 DEBUG [RS:1;jenkins-hbase4:41817] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 11:05:41,711 DEBUG [RS:0;jenkins-hbase4:34427] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 11:05:41,711 DEBUG [RS:1;jenkins-hbase4:41817] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 11:05:41,711 DEBUG [RS:0;jenkins-hbase4:34427] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 11:05:41,711 DEBUG [RS:0;jenkins-hbase4:34427] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 11:05:41,711 DEBUG [RS:0;jenkins-hbase4:34427] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:41,712 DEBUG [RS:0;jenkins-hbase4:34427] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34427,1689419140446' 2023-07-15 11:05:41,712 DEBUG [RS:0;jenkins-hbase4:34427] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 11:05:41,712 DEBUG [RS:1;jenkins-hbase4:41817] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 11:05:41,712 INFO [RS:1;jenkins-hbase4:41817] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 11:05:41,712 DEBUG [RS:0;jenkins-hbase4:34427] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 11:05:41,713 INFO [RS:1;jenkins-hbase4:41817] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 11:05:41,713 DEBUG [RS:0;jenkins-hbase4:34427] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 11:05:41,713 INFO [RS:0;jenkins-hbase4:34427] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 11:05:41,713 INFO [RS:0;jenkins-hbase4:34427] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 11:05:41,770 DEBUG [jenkins-hbase4:38773] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-15 11:05:41,774 DEBUG [jenkins-hbase4:38773] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 11:05:41,780 DEBUG [jenkins-hbase4:38773] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 11:05:41,780 DEBUG [jenkins-hbase4:38773] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 11:05:41,780 DEBUG [jenkins-hbase4:38773] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 11:05:41,783 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41817,1689419140491, state=OPENING 2023-07-15 11:05:41,790 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-15 11:05:41,792 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 11:05:41,792 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 11:05:41,797 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41817,1689419140491}] 2023-07-15 11:05:41,823 INFO [RS:0;jenkins-hbase4:34427] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34427%2C1689419140446, suffix=, logDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446, archiveDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/oldWALs, maxLogs=32 2023-07-15 11:05:41,823 INFO [RS:1;jenkins-hbase4:41817] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41817%2C1689419140491, suffix=, logDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491, archiveDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/oldWALs, maxLogs=32 2023-07-15 11:05:41,848 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34387,DS-e17b79f3-7f48-44a0-a435-8c634aa6dce9,DISK] 2023-07-15 11:05:41,849 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32961,DS-213137f8-2c0c-4882-b473-e4f5e74cacf4,DISK] 2023-07-15 11:05:41,849 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32961,DS-213137f8-2c0c-4882-b473-e4f5e74cacf4,DISK] 2023-07-15 11:05:41,850 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34387,DS-e17b79f3-7f48-44a0-a435-8c634aa6dce9,DISK] 2023-07-15 11:05:41,861 INFO [RS:1;jenkins-hbase4:41817] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491/jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825 2023-07-15 11:05:41,861 INFO [RS:0;jenkins-hbase4:34427] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446/jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825 2023-07-15 11:05:41,862 DEBUG [RS:1;jenkins-hbase4:41817] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34387,DS-e17b79f3-7f48-44a0-a435-8c634aa6dce9,DISK], DatanodeInfoWithStorage[127.0.0.1:32961,DS-213137f8-2c0c-4882-b473-e4f5e74cacf4,DISK]] 2023-07-15 11:05:41,863 DEBUG [RS:0;jenkins-hbase4:34427] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32961,DS-213137f8-2c0c-4882-b473-e4f5e74cacf4,DISK], DatanodeInfoWithStorage[127.0.0.1:34387,DS-e17b79f3-7f48-44a0-a435-8c634aa6dce9,DISK]] 2023-07-15 11:05:41,979 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,41817,1689419140491 2023-07-15 11:05:41,982 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 11:05:41,986 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 11:05:41,997 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-15 11:05:41,997 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 11:05:42,000 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta, suffix=.meta, logDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491, archiveDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/oldWALs, maxLogs=32 2023-07-15 11:05:42,017 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34387,DS-e17b79f3-7f48-44a0-a435-8c634aa6dce9,DISK] 2023-07-15 11:05:42,017 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32961,DS-213137f8-2c0c-4882-b473-e4f5e74cacf4,DISK] 2023-07-15 11:05:42,023 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491/jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta 2023-07-15 11:05:42,024 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34387,DS-e17b79f3-7f48-44a0-a435-8c634aa6dce9,DISK], DatanodeInfoWithStorage[127.0.0.1:32961,DS-213137f8-2c0c-4882-b473-e4f5e74cacf4,DISK]] 2023-07-15 11:05:42,024 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-15 11:05:42,026 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 11:05:42,044 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-15 11:05:42,071 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-15 11:05:42,076 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-15 11:05:42,076 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 11:05:42,077 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-15 11:05:42,077 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-15 11:05:42,079 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 11:05:42,081 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/info 2023-07-15 11:05:42,081 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/info 2023-07-15 11:05:42,082 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 11:05:42,083 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:05:42,083 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 11:05:42,085 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/rep_barrier 2023-07-15 11:05:42,085 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/rep_barrier 2023-07-15 11:05:42,085 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 11:05:42,086 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:05:42,086 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 11:05:42,090 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/table 2023-07-15 11:05:42,090 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/table 2023-07-15 11:05:42,090 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 11:05:42,091 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:05:42,093 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740 2023-07-15 11:05:42,096 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740 2023-07-15 11:05:42,100 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 11:05:42,102 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 11:05:42,103 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74534232, jitterRate=0.11064660549163818}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 11:05:42,104 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 11:05:42,113 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689419141972 2023-07-15 11:05:42,131 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-15 11:05:42,132 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-15 11:05:42,132 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41817,1689419140491, state=OPEN 2023-07-15 11:05:42,135 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 11:05:42,135 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 11:05:42,139 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-15 11:05:42,139 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41817,1689419140491 in 339 msec 2023-07-15 11:05:42,144 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-15 11:05:42,144 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 541 msec 2023-07-15 11:05:42,148 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 760 msec 2023-07-15 11:05:42,149 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689419142149, completionTime=-1 2023-07-15 11:05:42,149 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-15 11:05:42,150 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-15 11:05:42,216 DEBUG [hconnection-0x65e8c976-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 11:05:42,219 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46326, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 11:05:42,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-15 11:05:42,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689419202237 2023-07-15 11:05:42,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689419262237 2023-07-15 11:05:42,237 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 87 msec 2023-07-15 11:05:42,259 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38773,1689419139308-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:42,259 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38773,1689419139308-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:42,259 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38773,1689419139308-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:42,261 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:38773, period=300000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:42,261 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:42,267 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-15 11:05:42,275 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-15 11:05:42,277 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-15 11:05:42,287 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-15 11:05:42,289 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-15 11:05:42,292 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-15 11:05:42,316 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/.tmp/data/hbase/namespace/00c06350df1a1663c612751711e5cc66 2023-07-15 11:05:42,318 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/.tmp/data/hbase/namespace/00c06350df1a1663c612751711e5cc66 empty. 2023-07-15 11:05:42,319 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/.tmp/data/hbase/namespace/00c06350df1a1663c612751711e5cc66 2023-07-15 11:05:42,319 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-15 11:05:42,346 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-15 11:05:42,348 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 00c06350df1a1663c612751711e5cc66, NAME => 'hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/.tmp 2023-07-15 11:05:42,365 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 11:05:42,365 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 00c06350df1a1663c612751711e5cc66, disabling compactions & flushes 2023-07-15 11:05:42,365 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:05:42,365 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:05:42,365 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. after waiting 0 ms 2023-07-15 11:05:42,365 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:05:42,365 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:05:42,365 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 00c06350df1a1663c612751711e5cc66: 2023-07-15 11:05:42,369 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-15 11:05:42,384 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689419142372"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689419142372"}]},"ts":"1689419142372"} 2023-07-15 11:05:42,416 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-15 11:05:42,418 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-15 11:05:42,422 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689419142418"}]},"ts":"1689419142418"} 2023-07-15 11:05:42,426 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-15 11:05:42,431 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 11:05:42,432 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 11:05:42,432 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 11:05:42,432 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 11:05:42,434 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=00c06350df1a1663c612751711e5cc66, ASSIGN}] 2023-07-15 11:05:42,437 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=00c06350df1a1663c612751711e5cc66, ASSIGN 2023-07-15 11:05:42,439 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=00c06350df1a1663c612751711e5cc66, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34427,1689419140446; forceNewPlan=false, retain=false 2023-07-15 11:05:42,592 INFO [jenkins-hbase4:38773] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-15 11:05:42,593 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=00c06350df1a1663c612751711e5cc66, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:42,594 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689419142593"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689419142593"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689419142593"}]},"ts":"1689419142593"} 2023-07-15 11:05:42,598 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 00c06350df1a1663c612751711e5cc66, server=jenkins-hbase4.apache.org,34427,1689419140446}] 2023-07-15 11:05:42,752 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:42,753 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 11:05:42,755 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51962, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 11:05:42,761 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:05:42,762 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 00c06350df1a1663c612751711e5cc66, NAME => 'hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66.', STARTKEY => '', ENDKEY => ''} 2023-07-15 11:05:42,763 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 00c06350df1a1663c612751711e5cc66 2023-07-15 11:05:42,763 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 11:05:42,763 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 00c06350df1a1663c612751711e5cc66 2023-07-15 11:05:42,763 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 00c06350df1a1663c612751711e5cc66 2023-07-15 11:05:42,765 INFO [StoreOpener-00c06350df1a1663c612751711e5cc66-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 00c06350df1a1663c612751711e5cc66 2023-07-15 11:05:42,768 DEBUG [StoreOpener-00c06350df1a1663c612751711e5cc66-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/info 2023-07-15 11:05:42,768 DEBUG [StoreOpener-00c06350df1a1663c612751711e5cc66-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/info 2023-07-15 11:05:42,768 INFO [StoreOpener-00c06350df1a1663c612751711e5cc66-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 00c06350df1a1663c612751711e5cc66 columnFamilyName info 2023-07-15 11:05:42,769 INFO [StoreOpener-00c06350df1a1663c612751711e5cc66-1] regionserver.HStore(310): Store=00c06350df1a1663c612751711e5cc66/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:05:42,771 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66 2023-07-15 11:05:42,771 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66 2023-07-15 11:05:42,776 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 00c06350df1a1663c612751711e5cc66 2023-07-15 11:05:42,779 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 11:05:42,780 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 00c06350df1a1663c612751711e5cc66; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64074176, jitterRate=-0.045220375061035156}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 11:05:42,780 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 00c06350df1a1663c612751711e5cc66: 2023-07-15 11:05:42,782 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66., pid=6, masterSystemTime=1689419142752 2023-07-15 11:05:42,787 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:05:42,787 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:05:42,789 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=00c06350df1a1663c612751711e5cc66, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:42,789 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689419142788"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689419142788"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689419142788"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689419142788"}]},"ts":"1689419142788"} 2023-07-15 11:05:42,799 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-15 11:05:42,799 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 00c06350df1a1663c612751711e5cc66, server=jenkins-hbase4.apache.org,34427,1689419140446 in 197 msec 2023-07-15 11:05:42,802 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-15 11:05:42,803 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=00c06350df1a1663c612751711e5cc66, ASSIGN in 365 msec 2023-07-15 11:05:42,804 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-15 11:05:42,804 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689419142804"}]},"ts":"1689419142804"} 2023-07-15 11:05:42,807 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-15 11:05:42,810 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-15 11:05:42,813 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 533 msec 2023-07-15 11:05:42,890 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-15 11:05:42,891 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-15 11:05:42,891 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 11:05:42,913 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 11:05:42,917 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51976, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 11:05:42,931 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-15 11:05:42,950 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-15 11:05:42,955 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 31 msec 2023-07-15 11:05:42,964 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-15 11:05:42,973 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-15 11:05:42,978 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 13 msec 2023-07-15 11:05:42,990 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-15 11:05:42,993 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-15 11:05:42,993 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.446sec 2023-07-15 11:05:42,995 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-15 11:05:42,997 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-15 11:05:42,997 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-15 11:05:42,998 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38773,1689419139308-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-15 11:05:42,999 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38773,1689419139308-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-15 11:05:43,007 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-15 11:05:43,019 DEBUG [Listener at localhost/42421] zookeeper.ReadOnlyZKClient(139): Connect 0x38241035 to 127.0.0.1:63907 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 11:05:43,024 DEBUG [Listener at localhost/42421] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@cceefda, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 11:05:43,037 DEBUG [hconnection-0x455e9d85-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 11:05:43,048 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 11:05:43,058 INFO [Listener at localhost/42421] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,38773,1689419139308 2023-07-15 11:05:43,067 DEBUG [Listener at localhost/42421] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-15 11:05:43,069 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60892, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-15 11:05:43,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-15 11:05:43,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-15 11:05:43,085 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-15 11:05:43,086 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-15 11:05:43,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-15 11:05:43,089 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/.tmp/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:43,090 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/.tmp/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731 empty. 2023-07-15 11:05:43,092 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/.tmp/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:43,092 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-15 11:05:43,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-15 11:05:43,111 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-15 11:05:43,112 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => d233ca8b9c3d3c6dcc956b1b5ac83731, NAME => 't,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/.tmp 2023-07-15 11:05:43,128 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 11:05:43,128 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing d233ca8b9c3d3c6dcc956b1b5ac83731, disabling compactions & flushes 2023-07-15 11:05:43,128 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:43,128 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:43,128 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. after waiting 0 ms 2023-07-15 11:05:43,128 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:43,128 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:43,128 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for d233ca8b9c3d3c6dcc956b1b5ac83731: 2023-07-15 11:05:43,132 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-15 11:05:43,134 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689419143134"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689419143134"}]},"ts":"1689419143134"} 2023-07-15 11:05:43,137 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-15 11:05:43,138 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-15 11:05:43,138 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689419143138"}]},"ts":"1689419143138"} 2023-07-15 11:05:43,140 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-15 11:05:43,144 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-15 11:05:43,145 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-15 11:05:43,145 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-15 11:05:43,145 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-15 11:05:43,145 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=d233ca8b9c3d3c6dcc956b1b5ac83731, ASSIGN}] 2023-07-15 11:05:43,147 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=d233ca8b9c3d3c6dcc956b1b5ac83731, ASSIGN 2023-07-15 11:05:43,148 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=d233ca8b9c3d3c6dcc956b1b5ac83731, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41817,1689419140491; forceNewPlan=false, retain=false 2023-07-15 11:05:43,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-15 11:05:43,299 INFO [jenkins-hbase4:38773] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-15 11:05:43,300 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=d233ca8b9c3d3c6dcc956b1b5ac83731, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41817,1689419140491 2023-07-15 11:05:43,300 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689419143300"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689419143300"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689419143300"}]},"ts":"1689419143300"} 2023-07-15 11:05:43,303 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure d233ca8b9c3d3c6dcc956b1b5ac83731, server=jenkins-hbase4.apache.org,41817,1689419140491}] 2023-07-15 11:05:43,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-15 11:05:43,462 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:43,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d233ca8b9c3d3c6dcc956b1b5ac83731, NAME => 't,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.', STARTKEY => '', ENDKEY => ''} 2023-07-15 11:05:43,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:43,463 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 11:05:43,463 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:43,463 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:43,465 INFO [StoreOpener-d233ca8b9c3d3c6dcc956b1b5ac83731-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:43,467 DEBUG [StoreOpener-d233ca8b9c3d3c6dcc956b1b5ac83731-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/f 2023-07-15 11:05:43,467 DEBUG [StoreOpener-d233ca8b9c3d3c6dcc956b1b5ac83731-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/f 2023-07-15 11:05:43,467 INFO [StoreOpener-d233ca8b9c3d3c6dcc956b1b5ac83731-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d233ca8b9c3d3c6dcc956b1b5ac83731 columnFamilyName f 2023-07-15 11:05:43,468 INFO [StoreOpener-d233ca8b9c3d3c6dcc956b1b5ac83731-1] regionserver.HStore(310): Store=d233ca8b9c3d3c6dcc956b1b5ac83731/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:05:43,470 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:43,471 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:43,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:43,478 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-15 11:05:43,479 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d233ca8b9c3d3c6dcc956b1b5ac83731; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74842323, jitterRate=0.1152375191450119}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 11:05:43,479 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d233ca8b9c3d3c6dcc956b1b5ac83731: 2023-07-15 11:05:43,480 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731., pid=11, masterSystemTime=1689419143456 2023-07-15 11:05:43,482 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:43,482 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:43,483 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=d233ca8b9c3d3c6dcc956b1b5ac83731, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41817,1689419140491 2023-07-15 11:05:43,483 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689419143483"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689419143483"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689419143483"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689419143483"}]},"ts":"1689419143483"} 2023-07-15 11:05:43,489 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-15 11:05:43,489 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure d233ca8b9c3d3c6dcc956b1b5ac83731, server=jenkins-hbase4.apache.org,41817,1689419140491 in 183 msec 2023-07-15 11:05:43,492 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-15 11:05:43,493 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=d233ca8b9c3d3c6dcc956b1b5ac83731, ASSIGN in 345 msec 2023-07-15 11:05:43,493 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-15 11:05:43,494 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689419143494"}]},"ts":"1689419143494"} 2023-07-15 11:05:43,496 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-15 11:05:43,499 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-15 11:05:43,501 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 418 msec 2023-07-15 11:05:43,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-15 11:05:43,708 INFO [Listener at localhost/42421] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-15 11:05:43,709 DEBUG [Listener at localhost/42421] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-15 11:05:43,710 INFO [Listener at localhost/42421] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 11:05:43,714 INFO [Listener at localhost/42421] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-15 11:05:43,715 INFO [Listener at localhost/42421] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-15 11:05:43,715 INFO [Listener at localhost/42421] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-15 11:05:43,761 INFO [Listener at localhost/42421] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=295, OpenFileDescriptor=585, MaxFileDescriptor=60000, SystemLoadAverage=271, ProcessCount=171, AvailableMemoryMB=6358 2023-07-15 11:05:43,763 INFO [Listener at localhost/42421] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-15 11:05:43,767 DEBUG [Listener at localhost/42421] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-15 11:05:43,768 DEBUG [Listener at localhost/42421] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:43,768 INFO [Listener at localhost/42421] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-15 11:05:43,768 INFO [Listener at localhost/42421] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-15 11:05:43,771 INFO [Listener at localhost/42421] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689419142804/Put/vlen=2/seqid=0} 2023-07-15 11:05:43,772 INFO [Listener at localhost/42421] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-15 11:05:43,772 INFO [Listener at localhost/42421] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66./info:regioninfo/1689419142788/Put/vlen=41/seqid=0, hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66./info:seqnumDuringOpen/1689419142788/Put/vlen=8/seqid=0, hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66./info:server/1689419142788/Put/vlen=31/seqid=0, hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66./info:serverstartcode/1689419142788/Put/vlen=8/seqid=0, hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66./info:sn/1689419142593/Put/vlen=45/seqid=0, hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66./info:state/1689419142788/Put/vlen=4/seqid=0} 2023-07-15 11:05:43,772 INFO [Listener at localhost/42421] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 00c06350df1a1663c612751711e5cc66, NAME => 'hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66.', STARTKEY => '', ENDKEY => ''} 2023-07-15 11:05:43,772 INFO [Listener at localhost/42421] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689419143494/Put/vlen=2/seqid=0} 2023-07-15 11:05:43,773 INFO [Listener at localhost/42421] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-15 11:05:43,773 INFO [Listener at localhost/42421] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731./info:regioninfo/1689419143483/Put/vlen=35/seqid=0, t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731./info:seqnumDuringOpen/1689419143483/Put/vlen=8/seqid=0, t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731./info:server/1689419143483/Put/vlen=31/seqid=0, t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731./info:serverstartcode/1689419143483/Put/vlen=8/seqid=0, t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731./info:sn/1689419143300/Put/vlen=45/seqid=0, t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731./info:state/1689419143483/Put/vlen=4/seqid=0} 2023-07-15 11:05:43,773 INFO [Listener at localhost/42421] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => d233ca8b9c3d3c6dcc956b1b5ac83731, NAME => 't,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.', STARTKEY => '', ENDKEY => ''} 2023-07-15 11:05:43,774 DEBUG [Listener at localhost/42421] zookeeper.ReadOnlyZKClient(139): Connect 0x3e82be35 to 127.0.0.1:63907 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 11:05:43,779 DEBUG [Listener at localhost/42421] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@192ca9e4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 11:05:43,780 INFO [Listener at localhost/42421] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-15 11:05:43,780 INFO [Listener at localhost/42421] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-15 11:05:43,782 DEBUG [hconnection-0x4953d7ba-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 11:05:43,785 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46338, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 11:05:43,792 INFO [Listener at localhost/42421] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-15 11:05:43,795 ERROR [Listener at localhost/42421] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,41817,1689419140491: die! ***** 2023-07-15 11:05:43,795 ERROR [Listener at localhost/42421] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-15 11:05:43,799 DEBUG [Listener at localhost/42421] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-15 11:05:43,802 DEBUG [Listener at localhost/42421] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-15 11:05:43,802 DEBUG [Listener at localhost/42421] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-15 11:05:43,803 DEBUG [Listener at localhost/42421] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-15 11:05:43,804 INFO [Listener at localhost/42421] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 878182400, "init": 513802240, "max": 2051014656, "used": 397213784 }, "NonHeapMemoryUsage": { "committed": 113008640, "init": 2555904, "max": -1, "used": 110918976 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-15 11:05:43,808 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=38773] master.MasterRpcServices(612): jenkins-hbase4.apache.org,41817,1689419140491 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,41817,1689419140491: die! ***** 2023-07-15 11:05:43,811 INFO [Listener at localhost/42421] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,41817,1689419140491' ***** 2023-07-15 11:05:43,811 INFO [Listener at localhost/42421] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-15 11:05:43,811 INFO [RS:1;jenkins-hbase4:41817] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 11:05:43,812 INFO [RS:1;jenkins-hbase4:41817] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-15 11:05:43,812 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 11:05:43,812 INFO [RS:1;jenkins-hbase4:41817] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-15 11:05:43,812 INFO [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer(3305): Received CLOSE for d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:43,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41817] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner 4595329752172068875 2023-07-15 11:05:43,813 INFO [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,41817,1689419140491 2023-07-15 11:05:43,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41817] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41817,1689419140491 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 11:05:43,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d233ca8b9c3d3c6dcc956b1b5ac83731, disabling compactions & flushes 2023-07-15 11:05:43,813 DEBUG [RS:1;jenkins-hbase4:41817] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ff98d89 to 127.0.0.1:63907 2023-07-15 11:05:43,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41817] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:46338 deadline: 1689419203812, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41817,1689419140491 aborting 2023-07-15 11:05:43,814 DEBUG [RS:1;jenkins-hbase4:41817] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 11:05:43,813 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:43,814 INFO [RS:1;jenkins-hbase4:41817] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-15 11:05:43,814 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:43,814 INFO [RS:1;jenkins-hbase4:41817] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-15 11:05:43,814 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. after waiting 0 ms 2023-07-15 11:05:43,814 INFO [RS:1;jenkins-hbase4:41817] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-15 11:05:43,814 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:43,814 INFO [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-15 11:05:43,814 INFO [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-07-15 11:05:43,815 DEBUG [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, d233ca8b9c3d3c6dcc956b1b5ac83731=t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.} 2023-07-15 11:05:43,815 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:43,815 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 11:05:43,815 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 11:05:43,815 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 11:05:43,816 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:43,816 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 11:05:43,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d233ca8b9c3d3c6dcc956b1b5ac83731: 2023-07-15 11:05:43,816 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 11:05:43,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:43,817 DEBUG [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-15 11:05:43,817 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 2394 in region hbase:meta,,1.1588230740 2023-07-15 11:05:43,818 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 11:05:43,818 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 11:05:43,818 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 11:05:43,818 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-15 11:05:43,832 DEBUG [Listener at localhost/42421] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41817,1689419140491 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41817,1689419140491 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-15 11:05:43,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41817] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41817,1689419140491 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 11:05:43,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41817] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:46338 deadline: 1689419203862, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41817,1689419140491 aborting 2023-07-15 11:05:43,970 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=41817] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41817,1689419140491 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2513) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44992) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 11:05:43,970 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=41817] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:46338 deadline: 1689419203970, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41817,1689419140491 aborting 2023-07-15 11:05:44,018 INFO [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,41817,1689419140491; all regions closed. 2023-07-15 11:05:44,031 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491/jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825 not finished, retry = 0 2023-07-15 11:05:44,073 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=41817] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41817,1689419140491 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2513) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44992) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 11:05:44,073 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=41817] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:46338 deadline: 1689419204073, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41817,1689419140491 aborting 2023-07-15 11:05:44,134 DEBUG [RS:1;jenkins-hbase4:41817] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 11:05:44,135 INFO [RS:1;jenkins-hbase4:41817] regionserver.LeaseManager(133): Closed leases 2023-07-15 11:05:44,135 INFO [RS:1;jenkins-hbase4:41817] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-15 11:05:44,135 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 11:05:44,136 INFO [RS:1;jenkins-hbase4:41817] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41817 2023-07-15 11:05:44,145 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:41817-0x10168a8c89e0002, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41817,1689419140491 2023-07-15 11:05:44,145 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 11:05:44,145 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41817,1689419140491 2023-07-15 11:05:44,145 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:41817-0x10168a8c89e0002, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 11:05:44,145 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 11:05:44,147 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,41817,1689419140491] 2023-07-15 11:05:44,147 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,41817,1689419140491; numProcessing=1 2023-07-15 11:05:44,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:44,148 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,41817,1689419140491 znode expired, triggering replicatorRemoved event 2023-07-15 11:05:44,150 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,41817,1689419140491 already deleted, retry=false 2023-07-15 11:05:44,150 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,41817,1689419140491 on jenkins-hbase4.apache.org,38773,1689419139308 2023-07-15 11:05:44,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:44,157 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,41817,1689419140491, splitWal=true, meta=true 2023-07-15 11:05:44,157 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,41817,1689419140491 (carryingMeta=true) jenkins-hbase4.apache.org,41817,1689419140491/CRASHED/regionCount=2/lock=java.util.concurrent.locks.ReentrantReadWriteLock@4d03ac54[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-15 11:05:44,160 INFO [PEWorker-4] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,41817,1689419140491, splitWal=true, meta=true 2023-07-15 11:05:44,161 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,41817,1689419140491, splitWal=true, meta=true, isMeta: true 2023-07-15 11:05:44,163 DEBUG [PEWorker-4] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting 2023-07-15 11:05:44,164 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,41817,1689419140491 WAL count=1, meta=true 2023-07-15 11:05:44,167 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta}] 2023-07-15 11:05:44,175 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:44,176 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta, worker=jenkins-hbase4.apache.org,34427,1689419140446}] 2023-07-15 11:05:44,247 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:41817-0x10168a8c89e0002, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 11:05:44,248 INFO [RS:1;jenkins-hbase4:41817] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,41817,1689419140491; zookeeper connection closed. 2023-07-15 11:05:44,248 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:41817-0x10168a8c89e0002, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 11:05:44,248 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@57733dc1] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@57733dc1 2023-07-15 11:05:44,277 WARN [RPCClient-NioEventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:41817 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41817 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 11:05:44,278 DEBUG [RPCClient-NioEventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:41817 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41817 2023-07-15 11:05:44,340 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34427] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-15 11:05:44,361 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta, size=3.1 K (3185bytes) 2023-07-15 11:05:44,362 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta 2023-07-15 11:05:44,367 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta after 5ms 2023-07-15 11:05:44,372 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta took 12ms 2023-07-15 11:05:44,386 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-15 11:05:44,389 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta so closing down 2023-07-15 11:05:44,389 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 11:05:44,393 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-15 11:05:44,396 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta.temp 2023-07-15 11:05:44,397 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 11:05:44,409 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-15 11:05:44,411 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta.temp to hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-15 11:05:44,412 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 39 ms; skipped=1; WAL=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false 2023-07-15 11:05:44,412 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta, journal: Splitting hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta, size=3.1 K (3185bytes) at 1689419144362Finishing writing output for hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta so closing down at 1689419144389 (+27 ms)Creating recovered edits writer path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta.temp at 1689419144396 (+7 ms)3 split writer threads finished at 1689419144397 (+1 ms)Closed recovered edits writer path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689419144409 (+12 ms)Rename recovered edits hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta.temp to hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 at 1689419144412 (+3 ms)Processed 11 edits across 1 Regions in 39 ms; skipped=1; WAL=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false at 1689419144412 2023-07-15 11:05:44,412 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta 2023-07-15 11:05:44,415 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-15 11:05:44,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] master.HMaster(3641): Remote procedure done, pid=14 2023-07-15 11:05:44,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] wal.WALSplitUtil(141): Moved hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta to hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/oldWALs 2023-07-15 11:05:44,427 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-15 11:05:44,427 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta, worker=jenkins-hbase4.apache.org,34427,1689419140446 in 248 msec 2023-07-15 11:05:44,429 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:44,433 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-15 11:05:44,433 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C41817%2C1689419140491.meta.1689419142002.meta, worker=jenkins-hbase4.apache.org,34427,1689419140446 in 264 msec 2023-07-15 11:05:44,435 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting dir is empty, no logs to split. 2023-07-15 11:05:44,435 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,41817,1689419140491 WAL count=0, meta=true 2023-07-15 11:05:44,435 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,41817,1689419140491 WAL splitting is done? wals=0, meta=true 2023-07-15 11:05:44,436 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-15 11:05:44,437 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-15 11:05:44,438 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 11:05:44,580 DEBUG [hconnection-0x4953d7ba-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:41817 this server is in the failed servers list 2023-07-15 11:05:44,589 DEBUG [jenkins-hbase4:38773] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-15 11:05:44,589 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34427,1689419140446, state=OPENING 2023-07-15 11:05:44,590 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 11:05:44,591 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34427,1689419140446}] 2023-07-15 11:05:44,591 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 11:05:44,675 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 11:05:44,749 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-15 11:05:44,749 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 11:05:44,751 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta, suffix=.meta, logDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446, archiveDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/oldWALs, maxLogs=32 2023-07-15 11:05:44,770 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34387,DS-e17b79f3-7f48-44a0-a435-8c634aa6dce9,DISK] 2023-07-15 11:05:44,770 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32961,DS-213137f8-2c0c-4882-b473-e4f5e74cacf4,DISK] 2023-07-15 11:05:44,773 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446/jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta 2023-07-15 11:05:44,774 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34387,DS-e17b79f3-7f48-44a0-a435-8c634aa6dce9,DISK], DatanodeInfoWithStorage[127.0.0.1:32961,DS-213137f8-2c0c-4882-b473-e4f5e74cacf4,DISK]] 2023-07-15 11:05:44,774 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-15 11:05:44,774 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 11:05:44,774 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-15 11:05:44,775 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-15 11:05:44,775 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-15 11:05:44,775 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 11:05:44,775 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-15 11:05:44,775 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-15 11:05:44,777 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 11:05:44,778 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/info 2023-07-15 11:05:44,778 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/info 2023-07-15 11:05:44,779 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 11:05:44,779 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:05:44,779 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 11:05:44,781 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/rep_barrier 2023-07-15 11:05:44,781 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/rep_barrier 2023-07-15 11:05:44,781 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 11:05:44,782 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:05:44,782 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 11:05:44,783 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/table 2023-07-15 11:05:44,783 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/table 2023-07-15 11:05:44,783 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 11:05:44,784 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:05:44,785 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740 2023-07-15 11:05:44,789 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740 2023-07-15 11:05:44,790 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-15 11:05:44,798 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 24, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-15 11:05:44,798 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=2.34 KB heapSize=4.86 KB 2023-07-15 11:05:44,849 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.18 KB at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/.tmp/info/4bdfc82ad2d145e3985cd5bb4e44b6d5 2023-07-15 11:05:44,900 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/.tmp/table/4c32d0370a88451db571aaa82e5876f2 2023-07-15 11:05:44,908 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/.tmp/info/4bdfc82ad2d145e3985cd5bb4e44b6d5 as hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/info/4bdfc82ad2d145e3985cd5bb4e44b6d5 2023-07-15 11:05:44,916 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/info/4bdfc82ad2d145e3985cd5bb4e44b6d5, entries=20, sequenceid=13, filesize=6.9 K 2023-07-15 11:05:44,918 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/.tmp/table/4c32d0370a88451db571aaa82e5876f2 as hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/table/4c32d0370a88451db571aaa82e5876f2 2023-07-15 11:05:44,927 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/table/4c32d0370a88451db571aaa82e5876f2, entries=4, sequenceid=13, filesize=4.7 K 2023-07-15 11:05:44,927 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.34 KB/2394, heapSize ~4.58 KB/4688, currentSize=0 B/0 for 1588230740 in 129ms, sequenceid=13, compaction requested=false; wal=null 2023-07-15 11:05:44,928 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-15 11:05:44,931 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-15 11:05:44,934 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 11:05:44,936 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 11:05:44,939 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-15 11:05:44,940 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58814911, jitterRate=-0.12358953058719635}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 11:05:44,940 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 11:05:44,942 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=16, masterSystemTime=1689419144743 2023-07-15 11:05:44,944 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-15 11:05:44,944 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-15 11:05:44,945 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34427,1689419140446, state=OPEN 2023-07-15 11:05:44,946 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 11:05:44,946 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 11:05:44,949 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-15 11:05:44,949 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34427,1689419140446 in 356 msec 2023-07-15 11:05:44,952 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-15 11:05:44,952 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 513 msec 2023-07-15 11:05:44,952 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,41817,1689419140491 had 2 regions 2023-07-15 11:05:44,954 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,41817,1689419140491, splitWal=true, meta=true, isMeta: false 2023-07-15 11:05:44,956 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,41817,1689419140491 WAL count=1, meta=false 2023-07-15 11:05:44,956 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825}] 2023-07-15 11:05:44,958 DEBUG [PEWorker-1] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:44,959 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825, worker=jenkins-hbase4.apache.org,34427,1689419140446}] 2023-07-15 11:05:45,085 DEBUG [hconnection-0x4953d7ba-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 11:05:45,088 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51986, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 11:05:45,106 DEBUG [hconnection-0x4953d7ba-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:41817 this server is in the failed servers list 2023-07-15 11:05:45,112 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34427] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=18 2023-07-15 11:05:45,126 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825, size=1.1 K (1148bytes) 2023-07-15 11:05:45,126 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825 2023-07-15 11:05:45,126 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825 after 0ms 2023-07-15 11:05:45,130 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825 took 4ms 2023-07-15 11:05:45,133 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for d233ca8b9c3d3c6dcc956b1b5ac83731: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-15 11:05:45,135 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825 so closing down 2023-07-15 11:05:45,135 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 11:05:45,139 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825.temp 2023-07-15 11:05:45,139 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 11:05:45,150 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-15 11:05:45,153 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825.temp to hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/recovered.edits/0000000000000000013 2023-07-15 11:05:45,153 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 23 ms; skipped=1; WAL=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825, size=1.1 K, length=1148, corrupted=false, cancelled=false 2023-07-15 11:05:45,153 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825, journal: Splitting hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825, size=1.1 K (1148bytes) at 1689419145126Finishing writing output for hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825 so closing down at 1689419145135 (+9 ms)Creating recovered edits writer path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825.temp at 1689419145139 (+4 ms)3 split writer threads finished at 1689419145140 (+1 ms)Closed recovered edits writer path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689419145151 (+11 ms)Rename recovered edits hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825.temp to hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/recovered.edits/0000000000000000013 at 1689419145153 (+2 ms)Processed 11 edits across 1 Regions in 23 ms; skipped=1; WAL=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825, size=1.1 K, length=1148, corrupted=false, cancelled=false at 1689419145153 2023-07-15 11:05:45,154 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825 2023-07-15 11:05:45,154 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=18 2023-07-15 11:05:45,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] master.HMaster(3641): Remote procedure done, pid=18 2023-07-15 11:05:45,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] wal.WALSplitUtil(141): Moved hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting/jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825 to hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/oldWALs 2023-07-15 11:05:45,161 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-15 11:05:45,161 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825, worker=jenkins-hbase4.apache.org,34427,1689419140446 in 199 msec 2023-07-15 11:05:45,162 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:45,164 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=12 2023-07-15 11:05:45,164 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C41817%2C1689419140491.1689419141825, worker=jenkins-hbase4.apache.org,34427,1689419140446 in 206 msec 2023-07-15 11:05:45,166 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,41817,1689419140491-splitting dir is empty, no logs to split. 2023-07-15 11:05:45,166 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,41817,1689419140491 WAL count=0, meta=false 2023-07-15 11:05:45,166 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,41817,1689419140491 WAL splitting is done? wals=0, meta=false 2023-07-15 11:05:45,169 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=d233ca8b9c3d3c6dcc956b1b5ac83731, ASSIGN}] 2023-07-15 11:05:45,170 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=d233ca8b9c3d3c6dcc956b1b5ac83731, ASSIGN 2023-07-15 11:05:45,171 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=d233ca8b9c3d3c6dcc956b1b5ac83731, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 11:05:45,312 DEBUG [hconnection-0x4953d7ba-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:41817 this server is in the failed servers list 2023-07-15 11:05:45,321 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=d233ca8b9c3d3c6dcc956b1b5ac83731, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:45,321 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689419145321"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689419145321"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689419145321"}]},"ts":"1689419145321"} 2023-07-15 11:05:45,322 WARN [RS-EventLoopGroup-4-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:41817 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41817 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 11:05:45,323 DEBUG [RS-EventLoopGroup-4-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:41817 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41817 2023-07-15 11:05:45,428 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE; OpenRegionProcedure d233ca8b9c3d3c6dcc956b1b5ac83731, server=jenkins-hbase4.apache.org,34427,1689419140446}] 2023-07-15 11:05:45,586 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:45,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d233ca8b9c3d3c6dcc956b1b5ac83731, NAME => 't,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.', STARTKEY => '', ENDKEY => ''} 2023-07-15 11:05:45,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:45,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 11:05:45,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:45,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:45,588 INFO [StoreOpener-d233ca8b9c3d3c6dcc956b1b5ac83731-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:45,590 DEBUG [StoreOpener-d233ca8b9c3d3c6dcc956b1b5ac83731-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/f 2023-07-15 11:05:45,590 DEBUG [StoreOpener-d233ca8b9c3d3c6dcc956b1b5ac83731-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/f 2023-07-15 11:05:45,590 INFO [StoreOpener-d233ca8b9c3d3c6dcc956b1b5ac83731-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d233ca8b9c3d3c6dcc956b1b5ac83731 columnFamilyName f 2023-07-15 11:05:45,591 INFO [StoreOpener-d233ca8b9c3d3c6dcc956b1b5ac83731-1] regionserver.HStore(310): Store=d233ca8b9c3d3c6dcc956b1b5ac83731/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:05:45,593 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:45,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:45,596 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/recovered.edits/0000000000000000013 2023-07-15 11:05:45,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/recovered.edits/0000000000000000013 2023-07-15 11:05:45,600 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing d233ca8b9c3d3c6dcc956b1b5ac83731 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-15 11:05:45,614 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/.tmp/f/57021622e1104d4d84673cf74cc648fe 2023-07-15 11:05:45,620 DEBUG [hconnection-0x4953d7ba-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:41817 this server is in the failed servers list 2023-07-15 11:05:45,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/.tmp/f/57021622e1104d4d84673cf74cc648fe as hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/f/57021622e1104d4d84673cf74cc648fe 2023-07-15 11:05:45,631 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/f/57021622e1104d4d84673cf74cc648fe, entries=10, sequenceid=13, filesize=4.8 K 2023-07-15 11:05:45,631 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for d233ca8b9c3d3c6dcc956b1b5ac83731 in 31ms, sequenceid=13, compaction requested=false; wal=null 2023-07-15 11:05:45,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-15 11:05:45,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/recovered.edits/0000000000000000013 2023-07-15 11:05:45,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:05:45,642 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-15 11:05:45,643 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d233ca8b9c3d3c6dcc956b1b5ac83731; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72885799, jitterRate=0.08608303964138031}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 11:05:45,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d233ca8b9c3d3c6dcc956b1b5ac83731: 2023-07-15 11:05:45,645 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731., pid=20, masterSystemTime=1689419145580 2023-07-15 11:05:45,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:45,647 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:45,647 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=d233ca8b9c3d3c6dcc956b1b5ac83731, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:45,648 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689419145647"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689419145647"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689419145647"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689419145647"}]},"ts":"1689419145647"} 2023-07-15 11:05:45,653 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-07-15 11:05:45,653 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; OpenRegionProcedure d233ca8b9c3d3c6dcc956b1b5ac83731, server=jenkins-hbase4.apache.org,34427,1689419140446 in 222 msec 2023-07-15 11:05:45,655 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=12 2023-07-15 11:05:45,655 INFO [PEWorker-1] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,41817,1689419140491 after splitting done 2023-07-15 11:05:45,655 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=d233ca8b9c3d3c6dcc956b1b5ac83731, ASSIGN in 484 msec 2023-07-15 11:05:45,655 DEBUG [PEWorker-1] master.DeadServer(114): Removed jenkins-hbase4.apache.org,41817,1689419140491 from processing; numProcessing=0 2023-07-15 11:05:45,658 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,41817,1689419140491, splitWal=true, meta=true in 1.5040 sec 2023-07-15 11:05:46,135 DEBUG [Listener at localhost/42421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e82be35 to 127.0.0.1:63907 2023-07-15 11:05:46,135 DEBUG [Listener at localhost/42421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 11:05:46,135 INFO [Listener at localhost/42421] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-15 11:05:46,149 INFO [Listener at localhost/42421] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=268 (was 295), OpenFileDescriptor=569 (was 585), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=271 (was 271), ProcessCount=171 (was 171), AvailableMemoryMB=6302 (was 6358) 2023-07-15 11:05:46,160 INFO [Listener at localhost/42421] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=268, OpenFileDescriptor=569, MaxFileDescriptor=60000, SystemLoadAverage=271, ProcessCount=171, AvailableMemoryMB=6301 2023-07-15 11:05:46,160 INFO [Listener at localhost/42421] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase4:41817 2023-07-15 11:05:46,172 INFO [Listener at localhost/42421] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-15 11:05:46,172 INFO [Listener at localhost/42421] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 11:05:46,173 INFO [Listener at localhost/42421] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-15 11:05:46,173 INFO [Listener at localhost/42421] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-15 11:05:46,173 INFO [Listener at localhost/42421] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-15 11:05:46,173 INFO [Listener at localhost/42421] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-15 11:05:46,173 INFO [Listener at localhost/42421] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-15 11:05:46,174 INFO [Listener at localhost/42421] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39303 2023-07-15 11:05:46,175 INFO [Listener at localhost/42421] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-15 11:05:46,176 DEBUG [Listener at localhost/42421] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-15 11:05:46,176 INFO [Listener at localhost/42421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 11:05:46,177 INFO [Listener at localhost/42421] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-15 11:05:46,178 INFO [Listener at localhost/42421] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39303 connecting to ZooKeeper ensemble=127.0.0.1:63907 2023-07-15 11:05:46,183 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:393030x0, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-15 11:05:46,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:39303-0x10168a8c89e0008 connected 2023-07-15 11:05:46,184 DEBUG [Listener at localhost/42421] zookeeper.ZKUtil(162): regionserver:39303-0x10168a8c89e0008, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-15 11:05:46,185 DEBUG [Listener at localhost/42421] zookeeper.ZKUtil(162): regionserver:39303-0x10168a8c89e0008, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-15 11:05:46,186 DEBUG [Listener at localhost/42421] zookeeper.ZKUtil(164): regionserver:39303-0x10168a8c89e0008, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-15 11:05:46,188 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39303 2023-07-15 11:05:46,188 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39303 2023-07-15 11:05:46,188 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39303 2023-07-15 11:05:46,189 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39303 2023-07-15 11:05:46,189 DEBUG [Listener at localhost/42421] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39303 2023-07-15 11:05:46,191 INFO [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(951): ClusterId : c245dd99-d642-43ea-a0ee-16a79478bf9b 2023-07-15 11:05:46,191 DEBUG [RS:2;jenkins-hbase4:39303] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-15 11:05:46,193 DEBUG [RS:2;jenkins-hbase4:39303] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-15 11:05:46,193 DEBUG [RS:2;jenkins-hbase4:39303] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-15 11:05:46,195 DEBUG [RS:2;jenkins-hbase4:39303] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-15 11:05:46,196 DEBUG [RS:2;jenkins-hbase4:39303] zookeeper.ReadOnlyZKClient(139): Connect 0x38f07959 to 127.0.0.1:63907 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 11:05:46,205 DEBUG [RS:2;jenkins-hbase4:39303] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36379037, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 11:05:46,205 DEBUG [RS:2;jenkins-hbase4:39303] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32081229, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 11:05:46,213 DEBUG [RS:2;jenkins-hbase4:39303] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:39303 2023-07-15 11:05:46,214 INFO [RS:2;jenkins-hbase4:39303] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-15 11:05:46,214 INFO [RS:2;jenkins-hbase4:39303] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-15 11:05:46,214 DEBUG [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(1022): About to register with Master. 2023-07-15 11:05:46,214 INFO [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,38773,1689419139308 with isa=jenkins-hbase4.apache.org/172.31.14.131:39303, startcode=1689419146172 2023-07-15 11:05:46,214 DEBUG [RS:2;jenkins-hbase4:39303] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-15 11:05:46,218 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53037, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-15 11:05:46,218 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38773] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:05:46,219 DEBUG [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be 2023-07-15 11:05:46,219 DEBUG [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:33777 2023-07-15 11:05:46,219 DEBUG [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-15 11:05:46,221 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 11:05:46,221 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 11:05:46,222 DEBUG [RS:2;jenkins-hbase4:39303] zookeeper.ZKUtil(162): regionserver:39303-0x10168a8c89e0008, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:05:46,222 WARN [RS:2;jenkins-hbase4:39303] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-15 11:05:46,222 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39303,1689419146172] 2023-07-15 11:05:46,222 INFO [RS:2;jenkins-hbase4:39303] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 11:05:46,222 DEBUG [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(1948): logDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:05:46,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:05:46,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:46,226 DEBUG [RS:2;jenkins-hbase4:39303] zookeeper.ZKUtil(162): regionserver:39303-0x10168a8c89e0008, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:05:46,227 DEBUG [RS:2;jenkins-hbase4:39303] zookeeper.ZKUtil(162): regionserver:39303-0x10168a8c89e0008, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:05:46,228 DEBUG [RS:2;jenkins-hbase4:39303] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-15 11:05:46,228 INFO [RS:2;jenkins-hbase4:39303] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-15 11:05:46,231 INFO [RS:2;jenkins-hbase4:39303] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-15 11:05:46,234 INFO [RS:2;jenkins-hbase4:39303] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-15 11:05:46,235 INFO [RS:2;jenkins-hbase4:39303] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:46,237 INFO [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-15 11:05:46,239 INFO [RS:2;jenkins-hbase4:39303] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:46,239 DEBUG [RS:2;jenkins-hbase4:39303] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:46,239 DEBUG [RS:2;jenkins-hbase4:39303] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:46,239 DEBUG [RS:2;jenkins-hbase4:39303] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:46,240 DEBUG [RS:2;jenkins-hbase4:39303] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:46,240 DEBUG [RS:2;jenkins-hbase4:39303] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:46,240 DEBUG [RS:2;jenkins-hbase4:39303] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-15 11:05:46,240 DEBUG [RS:2;jenkins-hbase4:39303] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:46,240 DEBUG [RS:2;jenkins-hbase4:39303] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:46,240 DEBUG [RS:2;jenkins-hbase4:39303] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:46,240 DEBUG [RS:2;jenkins-hbase4:39303] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-15 11:05:46,242 INFO [RS:2;jenkins-hbase4:39303] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:46,242 INFO [RS:2;jenkins-hbase4:39303] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:46,242 INFO [RS:2;jenkins-hbase4:39303] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:46,256 INFO [RS:2;jenkins-hbase4:39303] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-15 11:05:46,256 INFO [RS:2;jenkins-hbase4:39303] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39303,1689419146172-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-15 11:05:46,267 INFO [RS:2;jenkins-hbase4:39303] regionserver.Replication(203): jenkins-hbase4.apache.org,39303,1689419146172 started 2023-07-15 11:05:46,267 INFO [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,39303,1689419146172, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39303, sessionid=0x10168a8c89e0008 2023-07-15 11:05:46,267 INFO [Listener at localhost/42421] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:39303,5,FailOnTimeoutGroup] 2023-07-15 11:05:46,267 DEBUG [RS:2;jenkins-hbase4:39303] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-15 11:05:46,267 DEBUG [RS:2;jenkins-hbase4:39303] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:05:46,267 INFO [Listener at localhost/42421] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,41817,1689419140491 2023-07-15 11:05:46,267 DEBUG [RS:2;jenkins-hbase4:39303] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39303,1689419146172' 2023-07-15 11:05:46,267 DEBUG [RS:2;jenkins-hbase4:39303] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-15 11:05:46,267 INFO [Listener at localhost/42421] client.TestScannerTimeout(177): START ************ test3686b 2023-07-15 11:05:46,268 DEBUG [RS:2;jenkins-hbase4:39303] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-15 11:05:46,268 DEBUG [RS:2;jenkins-hbase4:39303] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-15 11:05:46,268 DEBUG [RS:2;jenkins-hbase4:39303] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-15 11:05:46,268 DEBUG [RS:2;jenkins-hbase4:39303] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:05:46,268 DEBUG [RS:2;jenkins-hbase4:39303] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39303,1689419146172' 2023-07-15 11:05:46,268 DEBUG [RS:2;jenkins-hbase4:39303] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-15 11:05:46,269 WARN [RPCClient-NioEventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:41817 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41817 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 11:05:46,269 DEBUG [RS:2;jenkins-hbase4:39303] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-15 11:05:46,270 DEBUG [RPCClient-NioEventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:41817 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41817 2023-07-15 11:05:46,271 DEBUG [RS:2;jenkins-hbase4:39303] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-15 11:05:46,271 INFO [RS:2;jenkins-hbase4:39303] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-15 11:05:46,271 INFO [RS:2;jenkins-hbase4:39303] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-15 11:05:46,372 DEBUG [hconnection-0x455e9d85-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 11:05:46,374 INFO [RS:2;jenkins-hbase4:39303] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39303%2C1689419146172, suffix=, logDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,39303,1689419146172, archiveDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/oldWALs, maxLogs=32 2023-07-15 11:05:46,376 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51996, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 11:05:46,378 DEBUG [Listener at localhost/42421] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-15 11:05:46,378 DEBUG [Listener at localhost/42421] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:05:46,379 DEBUG [Listener at localhost/42421] zookeeper.ReadOnlyZKClient(139): Connect 0x486534a3 to 127.0.0.1:63907 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 11:05:46,384 DEBUG [Listener at localhost/42421] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52c9d026, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 11:05:46,387 DEBUG [hconnection-0x34dd6e0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 11:05:46,393 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52012, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 11:05:46,405 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32961,DS-213137f8-2c0c-4882-b473-e4f5e74cacf4,DISK] 2023-07-15 11:05:46,405 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34387,DS-e17b79f3-7f48-44a0-a435-8c634aa6dce9,DISK] 2023-07-15 11:05:46,407 INFO [RS:2;jenkins-hbase4:39303] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,39303,1689419146172/jenkins-hbase4.apache.org%2C39303%2C1689419146172.1689419146376 2023-07-15 11:05:46,411 DEBUG [RS:2;jenkins-hbase4:39303] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32961,DS-213137f8-2c0c-4882-b473-e4f5e74cacf4,DISK], DatanodeInfoWithStorage[127.0.0.1:34387,DS-e17b79f3-7f48-44a0-a435-8c634aa6dce9,DISK]] 2023-07-15 11:05:47,587 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-15 11:05:47,650 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-15 11:05:49,466 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 11:05:49,466 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-15 11:06:01,405 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease -8267774408001585143 expired clientIPAndPort=172.31.14.131:52012, userName=jenkins, regionInfo=t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:06:03,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34427] regionserver.RSRpcServices(3094): Client tried to access missing scanner -8267774408001585143 2023-07-15 11:06:03,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34427] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 172.31.14.131:52012 deadline: 1689419223407, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-8267774408001585143'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-15 11:06:03,409 DEBUG [Listener at localhost/42421] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-8267774408001585143'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-8267774408001585143'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-8267774408001585143'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-15 11:06:03,423 DEBUG [Listener at localhost/42421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x486534a3 to 127.0.0.1:63907 2023-07-15 11:06:03,423 DEBUG [Listener at localhost/42421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 11:06:03,423 INFO [Listener at localhost/42421] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-15 11:06:03,451 INFO [Listener at localhost/42421] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=296 (was 268) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_921456845_17 at /127.0.0.1:54076 [Receiving block BP-1750342079-172.31.14.131-1689419136325:blk_1073741845_1021] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63907@0x38f07959-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: PacketResponder: BP-1750342079-172.31.14.131-1689419136325:blk_1073741845_1021, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63907@0x38f07959 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/2035920984.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-1750342079-172.31.14.131-1689419136325:blk_1073741845_1021, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=39303 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63907@0x486534a3-SendThread(127.0.0.1:63907) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RS:2;jenkins-hbase4:39303 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63907@0x486534a3-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39303 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39303 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39303 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: jenkins-hbase4:39303Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:33777 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39303 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=39303 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63907@0x38f07959-SendThread(127.0.0.1:63907) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39303 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63907@0x486534a3 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/2035920984.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=39303 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_921456845_17 at /127.0.0.1:51534 [Receiving block BP-1750342079-172.31.14.131-1689419136325:blk_1073741845_1021] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/cluster_502b0317-3284-cd7f-c819-f42948f0d6d4/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/cluster_502b0317-3284-cd7f-c819-f42948f0d6d4/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:39303-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39303 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be-prefix:jenkins-hbase4.apache.org,39303,1689419146172 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=39303 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) - Thread LEAK? -, OpenFileDescriptor=573 (was 569) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=209 (was 271), ProcessCount=173 (was 171) - ProcessCount LEAK? -, AvailableMemoryMB=6744 (was 6301) - AvailableMemoryMB LEAK? - 2023-07-15 11:06:03,466 INFO [Listener at localhost/42421] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=296, OpenFileDescriptor=573, MaxFileDescriptor=60000, SystemLoadAverage=209, ProcessCount=173, AvailableMemoryMB=6737 2023-07-15 11:06:03,467 INFO [Listener at localhost/42421] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase4:41817 2023-07-15 11:06:03,467 INFO [Listener at localhost/42421] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,41817,1689419140491 2023-07-15 11:06:03,467 INFO [Listener at localhost/42421] client.TestScannerTimeout(104): START************ test2772 2023-07-15 11:06:03,471 DEBUG [Listener at localhost/42421] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-15 11:06:03,472 DEBUG [Listener at localhost/42421] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:06:03,472 DEBUG [Listener at localhost/42421] zookeeper.ReadOnlyZKClient(139): Connect 0x6208f527 to 127.0.0.1:63907 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-15 11:06:03,478 DEBUG [Listener at localhost/42421] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64cfe7c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-15 11:06:03,478 ERROR [Listener at localhost/42421] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,34427,1689419140446: die! ***** 2023-07-15 11:06:03,479 ERROR [Listener at localhost/42421] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-15 11:06:03,479 DEBUG [Listener at localhost/42421] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-15 11:06:03,480 DEBUG [Listener at localhost/42421] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-15 11:06:03,488 DEBUG [Listener at localhost/42421] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-15 11:06:03,493 DEBUG [Listener at localhost/42421] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-15 11:06:03,517 INFO [Listener at localhost/42421] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 982515712, "init": 513802240, "max": 2051014656, "used": 277166008 }, "NonHeapMemoryUsage": { "committed": 123756544, "init": 2555904, "max": -1, "used": 121397824 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 3, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 12789, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 4, "ProcessCallTime_num_ops": 63, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 63, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 63, "TotalCallTime_min": 1, "TotalCallTime_max": 1, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 63, "ResponseSize_min": 197, "ResponseSize_max": 679, "ResponseSize_mean": 438, "ResponseSize_25th_percentile": 199, "ResponseSize_median": 202, "ResponseSize_75th_percentile": 676, "ResponseSize_90th_percentile": 677, "ResponseSize_95th_percentile": 678, "ResponseSize_98th_percentile": 678, "ResponseSize_99th_percentile": 678, "ResponseSize_99.9th_percentile": 678, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 5, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 63, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 17495 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:63907", "tag.serverName": "jenkins-hbase4.apache.org,34427,1689419140446", "tag.clusterId": "c245dd99-d642-43ea-a0ee-16a79478bf9b", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 3, "maxStoreFileCount": 1, "memStoreSize": 912, "storeFileSize": 16785, "maxStoreFileAge": 16793, "minStoreFileAge": 16027, "avgStoreFileAge": 16520, "numReferenceFiles": 0, "regionServerStartTime": 1689419140446, "averageRegionSize": 5899, "storeFileIndexSize": 1048, "staticIndexSize": 184, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 1787, "localBytesRead": 1787, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819799248, "blockCacheCount": 3, "blockCacheDataBlockCount": 3, "blockCacheSize": 606640, "blockCacheCountHitPercent": 87.5, "blockCacheExpressHitPercent": 87.5, "l1CacheSize": 606640, "l1CacheFreeSize": 819799248, "l1CacheCount": 3, "l1CacheEvictionCount": 0, "l1CacheHitCount": 21, "l1CacheMissCount": 3, "l1CacheHitRatio": 0.875, "l1CacheMissRatio": 0.125, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 7076, "ByteBuffAllocatorPoolAllocationBytes": 1264640, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 24, "totalRowActionRequestCount": 30, "readRequestCount": 26, "filteredReadRequestCount": 0, "writeRequestCount": 4, "rpcGetRequestCount": 8, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 12, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 4, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 21, "blockCacheHitCountPrimary": 21, "blockCacheHitCachingCount": 21, "blockCacheMissCount": 3, "blockCacheMissCountPrimary": 3, "blockCacheMissCachingCount": 3, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 3, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 21, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 34, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 2664, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 11, "ScanTime_min": 0, "ScanTime_max": 1, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 4, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 2, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 8, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 16785, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 2, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 11, "ScanSize_min": 155, "ScanSize_max": 653, "ScanSize_mean": 404, "ScanSize_25th_percentile": 155, "ScanSize_median": 156, "ScanSize_75th_percentile": 652, "ScanSize_90th_percentile": 652, "ScanSize_95th_percentile": 652, "ScanSize_98th_percentile": 652, "ScanSize_99th_percentile": 652, "ScanSize_99.9th_percentile": 652, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 2664, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 2, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.470783869086365, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 24, "ServerWriteQueryPerSecond_mean_rate": 1.0981259916979083, "ServerWriteQueryPerSecond_1min_rate": 3.738243758742744, "ServerWriteQueryPerSecond_5min_rate": 4.565901237603426, "ServerWriteQueryPerSecond_15min_rate": 4.720662978343764 } ] } 2023-07-15 11:06:03,518 WARN [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=38773] master.MasterRpcServices(612): jenkins-hbase4.apache.org,34427,1689419140446 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,34427,1689419140446: die! ***** 2023-07-15 11:06:03,534 INFO [Listener at localhost/42421] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,34427,1689419140446' ***** 2023-07-15 11:06:03,537 INFO [Listener at localhost/42421] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-15 11:06:03,537 INFO [RS:0;jenkins-hbase4:34427] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 11:06:03,538 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 11:06:03,538 INFO [RS:0;jenkins-hbase4:34427] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-15 11:06:03,538 INFO [RS:0;jenkins-hbase4:34427] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-15 11:06:03,538 INFO [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(3305): Received CLOSE for 00c06350df1a1663c612751711e5cc66 2023-07-15 11:06:03,538 INFO [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(3305): Received CLOSE for d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:06:03,539 INFO [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:06:03,539 DEBUG [RS:0;jenkins-hbase4:34427] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29c90dff to 127.0.0.1:63907 2023-07-15 11:06:03,539 DEBUG [RS:0;jenkins-hbase4:34427] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 11:06:03,539 INFO [RS:0;jenkins-hbase4:34427] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-15 11:06:03,539 INFO [RS:0;jenkins-hbase4:34427] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-15 11:06:03,539 INFO [RS:0;jenkins-hbase4:34427] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-15 11:06:03,539 INFO [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-15 11:06:03,539 INFO [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-15 11:06:03,539 DEBUG [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(1478): Online Regions={00c06350df1a1663c612751711e5cc66=hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66., 1588230740=hbase:meta,,1.1588230740, d233ca8b9c3d3c6dcc956b1b5ac83731=t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.} 2023-07-15 11:06:03,539 DEBUG [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(1504): Waiting on 00c06350df1a1663c612751711e5cc66, 1588230740, d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:06:03,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 00c06350df1a1663c612751711e5cc66, disabling compactions & flushes 2023-07-15 11:06:03,541 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 11:06:03,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:06:03,541 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 11:06:03,541 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 11:06:03,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:06:03,541 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 11:06:03,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. after waiting 0 ms 2023-07-15 11:06:03,541 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 11:06:03,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:06:03,547 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:06:03,548 DEBUG [hconnection-0x6ecdfe5b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 11:06:03,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:06:03,549 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 00c06350df1a1663c612751711e5cc66: 2023-07-15 11:06:03,549 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:06:03,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d233ca8b9c3d3c6dcc956b1b5ac83731, disabling compactions & flushes 2023-07-15 11:06:03,553 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:06:03,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:06:03,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. after waiting 0 ms 2023-07-15 11:06:03,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:06:03,556 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49888, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 11:06:03,557 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34427] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,34427,1689419140446 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 11:06:03,557 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34427] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:49888 deadline: 1689419223557, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,34427,1689419140446 aborting 2023-07-15 11:06:03,563 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 834 in region hbase:meta,,1.1588230740 2023-07-15 11:06:03,563 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 11:06:03,563 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 11:06:03,563 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 11:06:03,564 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-15 11:06:03,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:06:03,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d233ca8b9c3d3c6dcc956b1b5ac83731: 2023-07-15 11:06:03,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:06:03,601 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 11:06:03,664 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34427] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,34427,1689419140446 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-15 11:06:03,665 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34427] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:49888 deadline: 1689419223664, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,34427,1689419140446 aborting 2023-07-15 11:06:03,740 INFO [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,34427,1689419140446; all regions closed. 2023-07-15 11:06:03,740 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-15 11:06:03,740 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-15 11:06:03,756 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446/jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825 not finished, retry = 0 2023-07-15 11:06:03,858 DEBUG [RS:0;jenkins-hbase4:34427] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 11:06:03,859 INFO [RS:0;jenkins-hbase4:34427] regionserver.LeaseManager(133): Closed leases 2023-07-15 11:06:03,859 INFO [RS:0;jenkins-hbase4:34427] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-15 11:06:03,859 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 11:06:03,860 INFO [RS:0;jenkins-hbase4:34427] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34427 2023-07-15 11:06:03,865 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:06:03,865 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 11:06:03,865 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 11:06:03,865 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:39303-0x10168a8c89e0008, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34427,1689419140446 2023-07-15 11:06:03,865 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:39303-0x10168a8c89e0008, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 11:06:03,867 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,34427,1689419140446] 2023-07-15 11:06:03,867 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34427,1689419140446; numProcessing=1 2023-07-15 11:06:03,868 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34427,1689419140446 already deleted, retry=false 2023-07-15 11:06:03,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39303-0x10168a8c89e0008, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:06:03,869 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:34427 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:34427 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 11:06:03,869 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,34427,1689419140446 znode expired, triggering replicatorRemoved event 2023-07-15 11:06:03,870 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:34427 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:34427 2023-07-15 11:06:03,869 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,34427,1689419140446 on jenkins-hbase4.apache.org,38773,1689419139308 2023-07-15 11:06:03,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39303-0x10168a8c89e0008, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:06:03,872 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=21, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,34427,1689419140446, splitWal=true, meta=true 2023-07-15 11:06:03,872 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=21 for jenkins-hbase4.apache.org,34427,1689419140446 (carryingMeta=true) jenkins-hbase4.apache.org,34427,1689419140446/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@407d2c28[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-15 11:06:03,874 INFO [PEWorker-2] procedure.ServerCrashProcedure(161): Start pid=21, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,34427,1689419140446, splitWal=true, meta=true 2023-07-15 11:06:03,875 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=21, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,34427,1689419140446, splitWal=true, meta=true, isMeta: true 2023-07-15 11:06:03,880 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting 2023-07-15 11:06:03,882 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,34427,1689419140446 WAL count=1, meta=true 2023-07-15 11:06:03,882 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta}] 2023-07-15 11:06:03,885 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:06:03,886 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta, worker=jenkins-hbase4.apache.org,39303,1689419146172}] 2023-07-15 11:06:03,967 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 11:06:03,967 INFO [RS:0;jenkins-hbase4:34427] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,34427,1689419140446; zookeeper connection closed. 2023-07-15 11:06:03,967 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:34427-0x10168a8c89e0001, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 11:06:03,968 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5201a8fd] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5201a8fd 2023-07-15 11:06:04,039 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:06:04,040 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-15 11:06:04,041 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-15 11:06:04,042 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39303] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=23 2023-07-15 11:06:04,058 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta, size=1.3 K (1341bytes) 2023-07-15 11:06:04,058 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta 2023-07-15 11:06:04,059 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta after 1ms 2023-07-15 11:06:04,064 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta took 6ms 2023-07-15 11:06:04,071 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 13 store_sequence_id { family_name: "info" sequence_id: 15 } store_sequence_id { family_name: "rep_barrier" sequence_id: 13 } store_sequence_id { family_name: "table" sequence_id: 13 } 2023-07-15 11:06:04,075 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta so closing down 2023-07-15 11:06:04,075 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 11:06:04,076 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta.temp 2023-07-15 11:06:04,076 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 11:06:04,086 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-15 11:06:04,088 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta.temp to hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-15 11:06:04,088 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 3 edits across 1 Regions in 21 ms; skipped=1; WAL=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta, size=1.3 K, length=1341, corrupted=false, cancelled=false 2023-07-15 11:06:04,088 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta, journal: Splitting hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta, size=1.3 K (1341bytes) at 1689419164058Finishing writing output for hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta so closing down at 1689419164075 (+17 ms)Creating recovered edits writer path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta.temp at 1689419164076 (+1 ms)3 split writer threads finished at 1689419164076Closed recovered edits writer path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689419164086 (+10 ms)Rename recovered edits hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta.temp to hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 at 1689419164088 (+2 ms)Processed 3 edits across 1 Regions in 21 ms; skipped=1; WAL=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta, size=1.3 K, length=1341, corrupted=false, cancelled=false at 1689419164088 2023-07-15 11:06:04,088 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta 2023-07-15 11:06:04,089 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=23 2023-07-15 11:06:04,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] master.HMaster(3641): Remote procedure done, pid=23 2023-07-15 11:06:04,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] wal.WALSplitUtil(141): Moved hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta to hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/oldWALs 2023-07-15 11:06:04,096 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-15 11:06:04,096 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta, worker=jenkins-hbase4.apache.org,39303,1689419146172 in 207 msec 2023-07-15 11:06:04,097 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:06:04,099 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-07-15 11:06:04,099 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C34427%2C1689419140446.meta.1689419144753.meta, worker=jenkins-hbase4.apache.org,39303,1689419146172 in 215 msec 2023-07-15 11:06:04,101 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting dir is empty, no logs to split. 2023-07-15 11:06:04,101 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,34427,1689419140446 WAL count=0, meta=true 2023-07-15 11:06:04,101 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,34427,1689419140446 WAL splitting is done? wals=0, meta=true 2023-07-15 11:06:04,102 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-15 11:06:04,103 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-15 11:06:04,104 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 11:06:04,171 DEBUG [hconnection-0x6ecdfe5b-metaLookup-shared--pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:34427 this server is in the failed servers list 2023-07-15 11:06:04,254 DEBUG [jenkins-hbase4:38773] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-15 11:06:04,255 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39303,1689419146172, state=OPENING 2023-07-15 11:06:04,260 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 11:06:04,261 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 11:06:04,261 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39303,1689419146172}] 2023-07-15 11:06:04,419 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-15 11:06:04,419 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-15 11:06:04,421 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39303%2C1689419146172.meta, suffix=.meta, logDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,39303,1689419146172, archiveDir=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/oldWALs, maxLogs=32 2023-07-15 11:06:04,437 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32961,DS-213137f8-2c0c-4882-b473-e4f5e74cacf4,DISK] 2023-07-15 11:06:04,438 DEBUG [RS-EventLoopGroup-6-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34387,DS-e17b79f3-7f48-44a0-a435-8c634aa6dce9,DISK] 2023-07-15 11:06:04,443 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,39303,1689419146172/jenkins-hbase4.apache.org%2C39303%2C1689419146172.meta.1689419164422.meta 2023-07-15 11:06:04,443 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32961,DS-213137f8-2c0c-4882-b473-e4f5e74cacf4,DISK], DatanodeInfoWithStorage[127.0.0.1:34387,DS-e17b79f3-7f48-44a0-a435-8c634aa6dce9,DISK]] 2023-07-15 11:06:04,443 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-15 11:06:04,443 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-15 11:06:04,444 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-15 11:06:04,444 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-15 11:06:04,444 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-15 11:06:04,444 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 11:06:04,444 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-15 11:06:04,444 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-15 11:06:04,446 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-15 11:06:04,447 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/info 2023-07-15 11:06:04,448 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/info 2023-07-15 11:06:04,448 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-15 11:06:04,462 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/info/4bdfc82ad2d145e3985cd5bb4e44b6d5 2023-07-15 11:06:04,463 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:06:04,463 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-15 11:06:04,465 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/rep_barrier 2023-07-15 11:06:04,465 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/rep_barrier 2023-07-15 11:06:04,466 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-15 11:06:04,467 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:06:04,467 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-15 11:06:04,470 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/table 2023-07-15 11:06:04,470 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/table 2023-07-15 11:06:04,471 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-15 11:06:04,484 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/table/4c32d0370a88451db571aaa82e5876f2 2023-07-15 11:06:04,484 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:06:04,485 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740 2023-07-15 11:06:04,488 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740 2023-07-15 11:06:04,489 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-15 11:06:04,493 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 8, skipped 0, firstSequenceIdInLog=16, maxSequenceIdInLog=17, path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-15 11:06:04,493 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=834 B heapSize=2.16 KB 2023-07-15 11:06:04,518 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=834 B at sequenceid=17 (bloomFilter=false), to=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/.tmp/info/cf2b1ed996214658abea10955fb69039 2023-07-15 11:06:04,528 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/.tmp/info/cf2b1ed996214658abea10955fb69039 as hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/info/cf2b1ed996214658abea10955fb69039 2023-07-15 11:06:04,536 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/info/cf2b1ed996214658abea10955fb69039, entries=8, sequenceid=17, filesize=5.5 K 2023-07-15 11:06:04,537 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~834 B/834, heapSize ~1.64 KB/1680, currentSize=0 B/0 for 1588230740 in 44ms, sequenceid=17, compaction requested=false; wal=null 2023-07-15 11:06:04,538 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-15 11:06:04,541 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-15 11:06:04,543 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-15 11:06:04,546 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=13 2023-07-15 11:06:04,547 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=18; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71352197, jitterRate=0.06323058903217316}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-15 11:06:04,547 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-15 11:06:04,549 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=25, masterSystemTime=1689419164413 2023-07-15 11:06:04,551 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-15 11:06:04,552 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-15 11:06:04,552 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39303,1689419146172, state=OPEN 2023-07-15 11:06:04,554 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-15 11:06:04,554 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-15 11:06:04,557 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-15 11:06:04,557 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39303,1689419146172 in 294 msec 2023-07-15 11:06:04,561 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=21 2023-07-15 11:06:04,561 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 456 msec 2023-07-15 11:06:04,561 INFO [PEWorker-1] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,34427,1689419140446 had 3 regions 2023-07-15 11:06:04,562 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=21, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,34427,1689419140446, splitWal=true, meta=true, isMeta: false 2023-07-15 11:06:04,565 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,34427,1689419140446 WAL count=1, meta=false 2023-07-15 11:06:04,565 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=21, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825}] 2023-07-15 11:06:04,568 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:06:04,569 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825, worker=jenkins-hbase4.apache.org,39303,1689419146172}] 2023-07-15 11:06:04,674 DEBUG [hconnection-0x6ecdfe5b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 11:06:04,677 INFO [RS-EventLoopGroup-6-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45086, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 11:06:04,685 DEBUG [hconnection-0x6ecdfe5b-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:34427 this server is in the failed servers list 2023-07-15 11:06:04,722 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39303] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=27 2023-07-15 11:06:04,736 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825, size=899 (899bytes) 2023-07-15 11:06:04,736 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825 2023-07-15 11:06:04,737 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825 after 1ms 2023-07-15 11:06:04,741 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825 took 5ms 2023-07-15 11:06:04,748 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 00c06350df1a1663c612751711e5cc66: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-15 11:06:04,751 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for d233ca8b9c3d3c6dcc956b1b5ac83731: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-15 11:06:04,751 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825 so closing down 2023-07-15 11:06:04,751 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-15 11:06:04,754 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825.temp 2023-07-15 11:06:04,754 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-15 11:06:04,774 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-15 11:06:04,776 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825.temp to hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/recovered.edits/0000000000000000005 2023-07-15 11:06:04,776 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 4 edits across 1 Regions in 31 ms; skipped=2; WAL=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825, size=899, length=899, corrupted=false, cancelled=false 2023-07-15 11:06:04,776 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825, journal: Splitting hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825, size=899 (899bytes) at 1689419164736Finishing writing output for hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825 so closing down at 1689419164751 (+15 ms)Creating recovered edits writer path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825.temp at 1689419164754 (+3 ms)3 split writer threads finished at 1689419164754Closed recovered edits writer path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689419164774 (+20 ms)Rename recovered edits hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825.temp to hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/recovered.edits/0000000000000000005 at 1689419164776 (+2 ms)Processed 4 edits across 1 Regions in 31 ms; skipped=2; WAL=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825, size=899, length=899, corrupted=false, cancelled=false at 1689419164776 2023-07-15 11:06:04,776 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825 2023-07-15 11:06:04,776 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=27 2023-07-15 11:06:04,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] master.HMaster(3641): Remote procedure done, pid=27 2023-07-15 11:06:04,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38773] wal.WALSplitUtil(141): Moved hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting/jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825 to hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/oldWALs 2023-07-15 11:06:04,784 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=26 2023-07-15 11:06:04,785 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825, worker=jenkins-hbase4.apache.org,39303,1689419146172 in 213 msec 2023-07-15 11:06:04,786 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:06:04,790 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=21 2023-07-15 11:06:04,790 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=21, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C34427%2C1689419140446.1689419141825, worker=jenkins-hbase4.apache.org,39303,1689419146172 in 221 msec 2023-07-15 11:06:04,792 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/WALs/jenkins-hbase4.apache.org,34427,1689419140446-splitting dir is empty, no logs to split. 2023-07-15 11:06:04,792 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,34427,1689419140446 WAL count=0, meta=false 2023-07-15 11:06:04,792 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,34427,1689419140446 WAL splitting is done? wals=0, meta=false 2023-07-15 11:06:04,794 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=00c06350df1a1663c612751711e5cc66, ASSIGN}, {pid=29, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=d233ca8b9c3d3c6dcc956b1b5ac83731, ASSIGN}] 2023-07-15 11:06:04,794 DEBUG [hconnection-0x6ecdfe5b-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:34427 this server is in the failed servers list 2023-07-15 11:06:04,795 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=00c06350df1a1663c612751711e5cc66, ASSIGN 2023-07-15 11:06:04,796 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=d233ca8b9c3d3c6dcc956b1b5ac83731, ASSIGN 2023-07-15 11:06:04,796 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=28, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=00c06350df1a1663c612751711e5cc66, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 11:06:04,796 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=d233ca8b9c3d3c6dcc956b1b5ac83731, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-15 11:06:04,947 DEBUG [jenkins-hbase4:38773] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-15 11:06:04,947 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=d233ca8b9c3d3c6dcc956b1b5ac83731, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:06:04,947 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=00c06350df1a1663c612751711e5cc66, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:06:04,947 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689419164947"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689419164947"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689419164947"}]},"ts":"1689419164947"} 2023-07-15 11:06:04,947 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689419164947"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689419164947"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689419164947"}]},"ts":"1689419164947"} 2023-07-15 11:06:04,948 WARN [RS-EventLoopGroup-4-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:34427 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:34427 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-15 11:06:04,949 DEBUG [RS-EventLoopGroup-4-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:34427 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:34427 2023-07-15 11:06:05,000 DEBUG [hconnection-0x6ecdfe5b-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:34427 this server is in the failed servers list 2023-07-15 11:06:05,051 DEBUG [PEWorker-3] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-15 11:06:05,052 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45102, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-15 11:06:05,054 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure d233ca8b9c3d3c6dcc956b1b5ac83731, server=jenkins-hbase4.apache.org,39303,1689419146172}] 2023-07-15 11:06:05,056 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=28, state=RUNNABLE; OpenRegionProcedure 00c06350df1a1663c612751711e5cc66, server=jenkins-hbase4.apache.org,39303,1689419146172}] 2023-07-15 11:06:05,217 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:06:05,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d233ca8b9c3d3c6dcc956b1b5ac83731, NAME => 't,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.', STARTKEY => '', ENDKEY => ''} 2023-07-15 11:06:05,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:06:05,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 11:06:05,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:06:05,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:06:05,224 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:06:05,225 INFO [StoreOpener-d233ca8b9c3d3c6dcc956b1b5ac83731-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:06:05,225 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 00c06350df1a1663c612751711e5cc66, NAME => 'hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66.', STARTKEY => '', ENDKEY => ''} 2023-07-15 11:06:05,225 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 00c06350df1a1663c612751711e5cc66 2023-07-15 11:06:05,225 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-15 11:06:05,225 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 00c06350df1a1663c612751711e5cc66 2023-07-15 11:06:05,225 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 00c06350df1a1663c612751711e5cc66 2023-07-15 11:06:05,226 DEBUG [StoreOpener-d233ca8b9c3d3c6dcc956b1b5ac83731-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/f 2023-07-15 11:06:05,226 DEBUG [StoreOpener-d233ca8b9c3d3c6dcc956b1b5ac83731-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/f 2023-07-15 11:06:05,227 INFO [StoreOpener-d233ca8b9c3d3c6dcc956b1b5ac83731-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d233ca8b9c3d3c6dcc956b1b5ac83731 columnFamilyName f 2023-07-15 11:06:05,227 INFO [StoreOpener-00c06350df1a1663c612751711e5cc66-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 00c06350df1a1663c612751711e5cc66 2023-07-15 11:06:05,228 DEBUG [StoreOpener-00c06350df1a1663c612751711e5cc66-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/info 2023-07-15 11:06:05,228 DEBUG [StoreOpener-00c06350df1a1663c612751711e5cc66-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/info 2023-07-15 11:06:05,228 INFO [StoreOpener-00c06350df1a1663c612751711e5cc66-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 00c06350df1a1663c612751711e5cc66 columnFamilyName info 2023-07-15 11:06:05,229 INFO [StoreOpener-00c06350df1a1663c612751711e5cc66-1] regionserver.HStore(310): Store=00c06350df1a1663c612751711e5cc66/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:06:05,230 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66 2023-07-15 11:06:05,233 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66 2023-07-15 11:06:05,233 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/recovered.edits/0000000000000000005 2023-07-15 11:06:05,235 DEBUG [StoreOpener-d233ca8b9c3d3c6dcc956b1b5ac83731-1] regionserver.HStore(539): loaded hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/f/57021622e1104d4d84673cf74cc648fe 2023-07-15 11:06:05,235 INFO [StoreOpener-d233ca8b9c3d3c6dcc956b1b5ac83731-1] regionserver.HStore(310): Store=d233ca8b9c3d3c6dcc956b1b5ac83731/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-15 11:06:05,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:06:05,238 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/recovered.edits/0000000000000000005 2023-07-15 11:06:05,238 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 00c06350df1a1663c612751711e5cc66 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-15 11:06:05,239 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:06:05,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:06:05,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-15 11:06:05,247 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d233ca8b9c3d3c6dcc956b1b5ac83731; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61555128, jitterRate=-0.08275711536407471}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 11:06:05,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d233ca8b9c3d3c6dcc956b1b5ac83731: 2023-07-15 11:06:05,248 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731., pid=30, masterSystemTime=1689419165207 2023-07-15 11:06:05,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:06:05,252 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:06:05,252 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=d233ca8b9c3d3c6dcc956b1b5ac83731, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:06:05,252 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689419165252"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689419165252"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689419165252"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689419165252"}]},"ts":"1689419165252"} 2023-07-15 11:06:05,262 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-07-15 11:06:05,263 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure d233ca8b9c3d3c6dcc956b1b5ac83731, server=jenkins-hbase4.apache.org,39303,1689419146172 in 200 msec 2023-07-15 11:06:05,264 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/.tmp/info/6b092821de9f41a398d7248613eb198b 2023-07-15 11:06:05,264 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=t, region=d233ca8b9c3d3c6dcc956b1b5ac83731, ASSIGN in 469 msec 2023-07-15 11:06:05,274 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/.tmp/info/6b092821de9f41a398d7248613eb198b as hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/info/6b092821de9f41a398d7248613eb198b 2023-07-15 11:06:05,281 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/info/6b092821de9f41a398d7248613eb198b, entries=2, sequenceid=5, filesize=4.8 K 2023-07-15 11:06:05,281 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 00c06350df1a1663c612751711e5cc66 in 43ms, sequenceid=5, compaction requested=false; wal=null 2023-07-15 11:06:05,282 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/recovered.edits/0000000000000000005 2023-07-15 11:06:05,286 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 00c06350df1a1663c612751711e5cc66 2023-07-15 11:06:05,288 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-15 11:06:05,290 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 00c06350df1a1663c612751711e5cc66; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74296539, jitterRate=0.10710470378398895}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-15 11:06:05,290 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 00c06350df1a1663c612751711e5cc66: 2023-07-15 11:06:05,291 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66., pid=31, masterSystemTime=1689419165207 2023-07-15 11:06:05,293 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:06:05,293 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:06:05,293 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=00c06350df1a1663c612751711e5cc66, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:06:05,294 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689419165293"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689419165293"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689419165293"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689419165293"}]},"ts":"1689419165293"} 2023-07-15 11:06:05,298 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=28 2023-07-15 11:06:05,298 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=28, state=SUCCESS; OpenRegionProcedure 00c06350df1a1663c612751711e5cc66, server=jenkins-hbase4.apache.org,39303,1689419146172 in 240 msec 2023-07-15 11:06:05,301 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=21 2023-07-15 11:06:05,301 INFO [PEWorker-4] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,34427,1689419140446 after splitting done 2023-07-15 11:06:05,301 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=00c06350df1a1663c612751711e5cc66, ASSIGN in 504 msec 2023-07-15 11:06:05,301 DEBUG [PEWorker-4] master.DeadServer(114): Removed jenkins-hbase4.apache.org,34427,1689419140446 from processing; numProcessing=0 2023-07-15 11:06:05,304 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=21, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,34427,1689419140446, splitWal=true, meta=true in 1.4310 sec 2023-07-15 11:06:05,311 DEBUG [Listener at localhost/42421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6208f527 to 127.0.0.1:63907 2023-07-15 11:06:05,311 DEBUG [Listener at localhost/42421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 11:06:05,311 INFO [Listener at localhost/42421] client.TestScannerTimeout(124): END ************ test2772 2023-07-15 11:06:05,325 INFO [Listener at localhost/42421] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=278 (was 296), OpenFileDescriptor=588 (was 573) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=209 (was 209), ProcessCount=173 (was 173), AvailableMemoryMB=6635 (was 6737) 2023-07-15 11:06:05,325 INFO [Listener at localhost/42421] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-15 11:06:05,326 INFO [Listener at localhost/42421] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-15 11:06:05,326 DEBUG [Listener at localhost/42421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38241035 to 127.0.0.1:63907 2023-07-15 11:06:05,326 DEBUG [Listener at localhost/42421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 11:06:05,326 DEBUG [Listener at localhost/42421] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-15 11:06:05,326 DEBUG [Listener at localhost/42421] util.JVMClusterUtil(257): Found active master hash=1752664830, stopped=false 2023-07-15 11:06:05,326 INFO [Listener at localhost/42421] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,38773,1689419139308 2023-07-15 11:06:05,331 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:39303-0x10168a8c89e0008, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-15 11:06:05,332 INFO [Listener at localhost/42421] procedure2.ProcedureExecutor(629): Stopping 2023-07-15 11:06:05,331 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-15 11:06:05,332 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 11:06:05,332 DEBUG [Listener at localhost/42421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x230e2d01 to 127.0.0.1:63907 2023-07-15 11:06:05,332 DEBUG [Listener at localhost/42421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 11:06:05,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39303-0x10168a8c89e0008, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 11:06:05,332 INFO [Listener at localhost/42421] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,39303,1689419146172' ***** 2023-07-15 11:06:05,332 INFO [Listener at localhost/42421] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-15 11:06:05,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-15 11:06:05,332 INFO [RS:2;jenkins-hbase4:39303] regionserver.HeapMemoryManager(220): Stopping 2023-07-15 11:06:05,333 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-15 11:06:05,333 INFO [RS:2;jenkins-hbase4:39303] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-15 11:06:05,333 INFO [RS:2;jenkins-hbase4:39303] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-15 11:06:05,333 INFO [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(3305): Received CLOSE for 00c06350df1a1663c612751711e5cc66 2023-07-15 11:06:05,333 INFO [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(3305): Received CLOSE for d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:06:05,333 INFO [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:06:05,333 DEBUG [RS:2;jenkins-hbase4:39303] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38f07959 to 127.0.0.1:63907 2023-07-15 11:06:05,334 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 00c06350df1a1663c612751711e5cc66, disabling compactions & flushes 2023-07-15 11:06:05,334 DEBUG [RS:2;jenkins-hbase4:39303] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 11:06:05,334 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:06:05,334 INFO [RS:2;jenkins-hbase4:39303] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-15 11:06:05,334 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:06:05,334 INFO [RS:2;jenkins-hbase4:39303] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-15 11:06:05,334 INFO [RS:2;jenkins-hbase4:39303] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-15 11:06:05,334 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. after waiting 0 ms 2023-07-15 11:06:05,334 INFO [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-15 11:06:05,334 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:06:05,334 INFO [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-15 11:06:05,334 DEBUG [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(1478): Online Regions={00c06350df1a1663c612751711e5cc66=hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66., 1588230740=hbase:meta,,1.1588230740, d233ca8b9c3d3c6dcc956b1b5ac83731=t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731.} 2023-07-15 11:06:05,335 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-15 11:06:05,335 DEBUG [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(1504): Waiting on 00c06350df1a1663c612751711e5cc66, 1588230740, d233ca8b9c3d3c6dcc956b1b5ac83731 2023-07-15 11:06:05,335 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-15 11:06:05,335 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-15 11:06:05,335 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-15 11:06:05,335 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-15 11:06:05,336 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-15 11:06:05,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/namespace/00c06350df1a1663c612751711e5cc66/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=5 2023-07-15 11:06:05,342 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:06:05,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 00c06350df1a1663c612751711e5cc66: 2023-07-15 11:06:05,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689419142276.00c06350df1a1663c612751711e5cc66. 2023-07-15 11:06:05,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d233ca8b9c3d3c6dcc956b1b5ac83731, disabling compactions & flushes 2023-07-15 11:06:05,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:06:05,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:06:05,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. after waiting 0 ms 2023-07-15 11:06:05,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:06:05,349 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/default/t/d233ca8b9c3d3c6dcc956b1b5ac83731/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-15 11:06:05,350 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:06:05,350 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d233ca8b9c3d3c6dcc956b1b5ac83731: 2023-07-15 11:06:05,350 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689419143077.d233ca8b9c3d3c6dcc956b1b5ac83731. 2023-07-15 11:06:05,363 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/.tmp/info/5475a7ab01154c048cf5da4c59b54940 2023-07-15 11:06:05,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/.tmp/info/5475a7ab01154c048cf5da4c59b54940 as hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/info/5475a7ab01154c048cf5da4c59b54940 2023-07-15 11:06:05,377 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/info/5475a7ab01154c048cf5da4c59b54940, entries=16, sequenceid=24, filesize=6.4 K 2023-07-15 11:06:05,378 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 43ms, sequenceid=24, compaction requested=true 2023-07-15 11:06:05,385 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=17 2023-07-15 11:06:05,386 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-15 11:06:05,386 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-15 11:06:05,386 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-15 11:06:05,386 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-15 11:06:05,535 INFO [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,39303,1689419146172; all regions closed. 2023-07-15 11:06:05,542 DEBUG [RS:2;jenkins-hbase4:39303] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/oldWALs 2023-07-15 11:06:05,542 INFO [RS:2;jenkins-hbase4:39303] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39303%2C1689419146172.meta:.meta(num 1689419164422) 2023-07-15 11:06:05,547 DEBUG [RS:2;jenkins-hbase4:39303] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/oldWALs 2023-07-15 11:06:05,547 INFO [RS:2;jenkins-hbase4:39303] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39303%2C1689419146172:(num 1689419146376) 2023-07-15 11:06:05,547 DEBUG [RS:2;jenkins-hbase4:39303] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 11:06:05,547 INFO [RS:2;jenkins-hbase4:39303] regionserver.LeaseManager(133): Closed leases 2023-07-15 11:06:05,547 INFO [RS:2;jenkins-hbase4:39303] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-15 11:06:05,547 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 11:06:05,548 INFO [RS:2;jenkins-hbase4:39303] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39303 2023-07-15 11:06:05,550 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:39303-0x10168a8c89e0008, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39303,1689419146172 2023-07-15 11:06:05,550 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 11:06:05,550 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:39303-0x10168a8c89e0008, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-15 11:06:05,552 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39303,1689419146172] 2023-07-15 11:06:05,552 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39303,1689419146172; numProcessing=1 2023-07-15 11:06:05,553 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39303,1689419146172 already deleted, retry=false 2023-07-15 11:06:05,553 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,39303,1689419146172 expired; onlineServers=0 2023-07-15 11:06:05,554 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,38773,1689419139308' ***** 2023-07-15 11:06:05,554 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-15 11:06:05,554 DEBUG [M:0;jenkins-hbase4:38773] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@294a80c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-15 11:06:05,554 INFO [M:0;jenkins-hbase4:38773] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,38773,1689419139308 2023-07-15 11:06:05,554 INFO [M:0;jenkins-hbase4:38773] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,38773,1689419139308; all regions closed. 2023-07-15 11:06:05,554 DEBUG [M:0;jenkins-hbase4:38773] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-15 11:06:05,554 DEBUG [M:0;jenkins-hbase4:38773] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-15 11:06:05,555 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-15 11:06:05,555 DEBUG [M:0;jenkins-hbase4:38773] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-15 11:06:05,555 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689419141507] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689419141507,5,FailOnTimeoutGroup] 2023-07-15 11:06:05,555 INFO [M:0;jenkins-hbase4:38773] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-15 11:06:05,555 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689419141507] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689419141507,5,FailOnTimeoutGroup] 2023-07-15 11:06:05,555 INFO [M:0;jenkins-hbase4:38773] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-15 11:06:05,555 INFO [M:0;jenkins-hbase4:38773] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-15 11:06:05,556 DEBUG [M:0;jenkins-hbase4:38773] master.HMaster(1512): Stopping service threads 2023-07-15 11:06:05,556 INFO [M:0;jenkins-hbase4:38773] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-15 11:06:05,556 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-15 11:06:05,556 ERROR [M:0;jenkins-hbase4:38773] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (1453727570) connection to localhost/127.0.0.1:33777 from jenkins,5,PEWorkerGroup] 2023-07-15 11:06:05,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-15 11:06:05,556 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-15 11:06:05,557 INFO [M:0;jenkins-hbase4:38773] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-15 11:06:05,557 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-15 11:06:05,557 DEBUG [M:0;jenkins-hbase4:38773] zookeeper.ZKUtil(398): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-15 11:06:05,557 WARN [M:0;jenkins-hbase4:38773] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-15 11:06:05,557 INFO [M:0;jenkins-hbase4:38773] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-15 11:06:05,558 INFO [M:0;jenkins-hbase4:38773] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-15 11:06:05,558 DEBUG [M:0;jenkins-hbase4:38773] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-15 11:06:05,558 INFO [M:0;jenkins-hbase4:38773] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 11:06:05,558 DEBUG [M:0;jenkins-hbase4:38773] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 11:06:05,558 DEBUG [M:0;jenkins-hbase4:38773] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-15 11:06:05,558 DEBUG [M:0;jenkins-hbase4:38773] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 11:06:05,558 INFO [M:0;jenkins-hbase4:38773] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=105.76 KB heapSize=127.45 KB 2023-07-15 11:06:05,575 INFO [M:0;jenkins-hbase4:38773] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=105.76 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/97d8964fbc6546af8cf900e2bfa9517c 2023-07-15 11:06:05,582 DEBUG [M:0;jenkins-hbase4:38773] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/97d8964fbc6546af8cf900e2bfa9517c as hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/97d8964fbc6546af8cf900e2bfa9517c 2023-07-15 11:06:05,588 INFO [M:0;jenkins-hbase4:38773] regionserver.HStore(1080): Added hdfs://localhost:33777/user/jenkins/test-data/ac71fcd1-0fda-26c5-12df-ed19513da3be/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/97d8964fbc6546af8cf900e2bfa9517c, entries=31, sequenceid=244, filesize=8.4 K 2023-07-15 11:06:05,589 INFO [M:0;jenkins-hbase4:38773] regionserver.HRegion(2948): Finished flush of dataSize ~105.76 KB/108302, heapSize ~127.44 KB/130496, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 31ms, sequenceid=244, compaction requested=false 2023-07-15 11:06:05,591 INFO [M:0;jenkins-hbase4:38773] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-15 11:06:05,591 DEBUG [M:0;jenkins-hbase4:38773] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-15 11:06:05,594 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-15 11:06:05,594 INFO [M:0;jenkins-hbase4:38773] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-15 11:06:05,595 INFO [M:0;jenkins-hbase4:38773] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38773 2023-07-15 11:06:05,596 DEBUG [M:0;jenkins-hbase4:38773] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,38773,1689419139308 already deleted, retry=false 2023-07-15 11:06:05,815 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 11:06:05,815 INFO [M:0;jenkins-hbase4:38773] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,38773,1689419139308; zookeeper connection closed. 2023-07-15 11:06:05,815 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): master:38773-0x10168a8c89e0000, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 11:06:05,915 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:39303-0x10168a8c89e0008, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 11:06:05,916 INFO [RS:2;jenkins-hbase4:39303] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,39303,1689419146172; zookeeper connection closed. 2023-07-15 11:06:05,916 DEBUG [Listener at localhost/42421-EventThread] zookeeper.ZKWatcher(600): regionserver:39303-0x10168a8c89e0008, quorum=127.0.0.1:63907, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-15 11:06:05,916 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@11c88e34] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@11c88e34 2023-07-15 11:06:05,916 INFO [Listener at localhost/42421] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-15 11:06:05,917 WARN [Listener at localhost/42421] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-15 11:06:05,920 INFO [Listener at localhost/42421] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-15 11:06:06,025 WARN [BP-1750342079-172.31.14.131-1689419136325 heartbeating to localhost/127.0.0.1:33777] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-15 11:06:06,025 WARN [BP-1750342079-172.31.14.131-1689419136325 heartbeating to localhost/127.0.0.1:33777] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1750342079-172.31.14.131-1689419136325 (Datanode Uuid cda3437d-c0d2-4e8a-a7b3-d735ed597d37) service to localhost/127.0.0.1:33777 2023-07-15 11:06:06,027 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/cluster_502b0317-3284-cd7f-c819-f42948f0d6d4/dfs/data/data3/current/BP-1750342079-172.31.14.131-1689419136325] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 11:06:06,027 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/cluster_502b0317-3284-cd7f-c819-f42948f0d6d4/dfs/data/data4/current/BP-1750342079-172.31.14.131-1689419136325] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 11:06:06,028 WARN [Listener at localhost/42421] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-15 11:06:06,030 INFO [Listener at localhost/42421] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-15 11:06:06,136 WARN [BP-1750342079-172.31.14.131-1689419136325 heartbeating to localhost/127.0.0.1:33777] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-15 11:06:06,136 WARN [BP-1750342079-172.31.14.131-1689419136325 heartbeating to localhost/127.0.0.1:33777] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1750342079-172.31.14.131-1689419136325 (Datanode Uuid 0f02e0a7-cac3-493c-8997-9fd577754c46) service to localhost/127.0.0.1:33777 2023-07-15 11:06:06,137 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/cluster_502b0317-3284-cd7f-c819-f42948f0d6d4/dfs/data/data1/current/BP-1750342079-172.31.14.131-1689419136325] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 11:06:06,137 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/89468615-f827-4c3e-167c-e2792487088c/cluster_502b0317-3284-cd7f-c819-f42948f0d6d4/dfs/data/data2/current/BP-1750342079-172.31.14.131-1689419136325] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-15 11:06:06,175 INFO [Listener at localhost/42421] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-15 11:06:06,244 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-15 11:06:06,295 INFO [Listener at localhost/42421] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-15 11:06:06,339 INFO [Listener at localhost/42421] hbase.HBaseTestingUtility(1293): Minicluster is down