2023-05-22 03:52:58,265 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366 2023-05-22 03:52:58,278 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.regionserver.wal.TestLogRolling timeout: 13 mins 2023-05-22 03:52:58,308 INFO [Time-limited test] hbase.ResourceChecker(147): before: regionserver.wal.TestLogRolling#testSlowSyncLogRolling Thread=10, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=384, ProcessCount=168, AvailableMemoryMB=8589 2023-05-22 03:52:58,317 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-05-22 03:52:58,318 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/cluster_ffd434c3-71a0-71a8-8706-dca49e1cb782, deleteOnExit=true 2023-05-22 03:52:58,318 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-05-22 03:52:58,319 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/test.cache.data in system properties and HBase conf 2023-05-22 03:52:58,320 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/hadoop.tmp.dir in system properties and HBase conf 2023-05-22 03:52:58,320 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/hadoop.log.dir in system properties and HBase conf 2023-05-22 03:52:58,321 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/mapreduce.cluster.local.dir in system properties and HBase conf 2023-05-22 03:52:58,321 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-05-22 03:52:58,321 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-05-22 03:52:58,425 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-05-22 03:52:58,777 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-05-22 03:52:58,781 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-05-22 03:52:58,781 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-05-22 03:52:58,782 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-05-22 03:52:58,782 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-05-22 03:52:58,782 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-05-22 03:52:58,782 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-05-22 03:52:58,783 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-05-22 03:52:58,783 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/dfs.journalnode.edits.dir in system properties and HBase conf 2023-05-22 03:52:58,784 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-05-22 03:52:58,784 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/nfs.dump.dir in system properties and HBase conf 2023-05-22 03:52:58,784 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/java.io.tmpdir in system properties and HBase conf 2023-05-22 03:52:58,785 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/dfs.journalnode.edits.dir in system properties and HBase conf 2023-05-22 03:52:58,785 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-05-22 03:52:58,785 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-05-22 03:52:59,284 WARN [Time-limited test] blockmanagement.DatanodeManager(362): The given interval for marking stale datanode = 30000, which is larger than heartbeat expire interval 20000. 2023-05-22 03:52:59,298 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(1) assuming SECONDS 2023-05-22 03:52:59,302 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-05-22 03:52:59,561 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-05-22 03:52:59,734 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-05-22 03:52:59,753 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:52:59,800 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:52:59,846 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/java.io.tmpdir/Jetty_localhost_localdomain_41279_hdfs____.issjsf/webapp 2023-05-22 03:53:00,049 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:41279 2023-05-22 03:53:00,060 WARN [Time-limited test] blockmanagement.DatanodeManager(362): The given interval for marking stale datanode = 30000, which is larger than heartbeat expire interval 20000. 2023-05-22 03:53:00,063 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(1) assuming SECONDS 2023-05-22 03:53:00,063 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-05-22 03:53:00,455 WARN [Listener at localhost.localdomain/36103] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:53:00,511 WARN [Listener at localhost.localdomain/36103] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:53:00,526 WARN [Listener at localhost.localdomain/36103] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:53:00,535 INFO [Listener at localhost.localdomain/36103] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:53:00,541 INFO [Listener at localhost.localdomain/36103] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/java.io.tmpdir/Jetty_localhost_38593_datanode____t3i92p/webapp 2023-05-22 03:53:00,616 INFO [Listener at localhost.localdomain/36103] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38593 2023-05-22 03:53:00,862 WARN [Listener at localhost.localdomain/40865] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:53:00,869 WARN [Listener at localhost.localdomain/40865] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:53:00,874 WARN [Listener at localhost.localdomain/40865] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:53:00,876 INFO [Listener at localhost.localdomain/40865] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:53:00,881 INFO [Listener at localhost.localdomain/40865] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/java.io.tmpdir/Jetty_localhost_44973_datanode____.be1ow4/webapp 2023-05-22 03:53:00,957 INFO [Listener at localhost.localdomain/40865] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44973 2023-05-22 03:53:00,965 WARN [Listener at localhost.localdomain/33637] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:53:01,219 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x78e546789f160c9b: Processing first storage report for DS-65aa84e3-834a-48ba-9786-b533e62f8b4b from datanode 9f31610a-e069-4fa6-854b-5af1903b9198 2023-05-22 03:53:01,221 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x78e546789f160c9b: from storage DS-65aa84e3-834a-48ba-9786-b533e62f8b4b node DatanodeRegistration(127.0.0.1:44901, datanodeUuid=9f31610a-e069-4fa6-854b-5af1903b9198, infoPort=40537, infoSecurePort=0, ipcPort=40865, storageInfo=lv=-57;cid=testClusterID;nsid=681979494;c=1684727579365), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-05-22 03:53:01,221 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5814e74ed1f3923f: Processing first storage report for DS-149d08fb-a4de-4c4a-8397-01671b74caa8 from datanode 180eb67b-bdae-4f32-a969-3e7d24f5e7ac 2023-05-22 03:53:01,221 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5814e74ed1f3923f: from storage DS-149d08fb-a4de-4c4a-8397-01671b74caa8 node DatanodeRegistration(127.0.0.1:39671, datanodeUuid=180eb67b-bdae-4f32-a969-3e7d24f5e7ac, infoPort=42475, infoSecurePort=0, ipcPort=33637, storageInfo=lv=-57;cid=testClusterID;nsid=681979494;c=1684727579365), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-05-22 03:53:01,221 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x78e546789f160c9b: Processing first storage report for DS-0e884aba-0fa6-4b27-bfe9-dfcfa076d045 from datanode 9f31610a-e069-4fa6-854b-5af1903b9198 2023-05-22 03:53:01,221 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x78e546789f160c9b: from storage DS-0e884aba-0fa6-4b27-bfe9-dfcfa076d045 node DatanodeRegistration(127.0.0.1:44901, datanodeUuid=9f31610a-e069-4fa6-854b-5af1903b9198, infoPort=40537, infoSecurePort=0, ipcPort=40865, storageInfo=lv=-57;cid=testClusterID;nsid=681979494;c=1684727579365), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:53:01,221 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5814e74ed1f3923f: Processing first storage report for DS-03c34ae2-b99c-48b6-b81a-e7502058f4ce from datanode 180eb67b-bdae-4f32-a969-3e7d24f5e7ac 2023-05-22 03:53:01,222 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5814e74ed1f3923f: from storage DS-03c34ae2-b99c-48b6-b81a-e7502058f4ce node DatanodeRegistration(127.0.0.1:39671, datanodeUuid=180eb67b-bdae-4f32-a969-3e7d24f5e7ac, infoPort=42475, infoSecurePort=0, ipcPort=33637, storageInfo=lv=-57;cid=testClusterID;nsid=681979494;c=1684727579365), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:53:01,304 DEBUG [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366 2023-05-22 03:53:01,359 INFO [Listener at localhost.localdomain/33637] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/cluster_ffd434c3-71a0-71a8-8706-dca49e1cb782/zookeeper_0, clientPort=58845, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/cluster_ffd434c3-71a0-71a8-8706-dca49e1cb782/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/cluster_ffd434c3-71a0-71a8-8706-dca49e1cb782/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-05-22 03:53:01,370 INFO [Listener at localhost.localdomain/33637] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58845 2023-05-22 03:53:01,377 INFO [Listener at localhost.localdomain/33637] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:53:01,379 INFO [Listener at localhost.localdomain/33637] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:53:01,985 INFO [Listener at localhost.localdomain/33637] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918 with version=8 2023-05-22 03:53:01,985 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/hbase-staging 2023-05-22 03:53:02,223 INFO [Listener at localhost.localdomain/33637] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-22 03:53:02,581 INFO [Listener at localhost.localdomain/33637] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-05-22 03:53:02,606 INFO [Listener at localhost.localdomain/33637] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:53:02,606 INFO [Listener at localhost.localdomain/33637] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-05-22 03:53:02,607 INFO [Listener at localhost.localdomain/33637] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-05-22 03:53:02,607 INFO [Listener at localhost.localdomain/33637] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:53:02,607 INFO [Listener at localhost.localdomain/33637] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-05-22 03:53:02,718 INFO [Listener at localhost.localdomain/33637] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-05-22 03:53:02,776 DEBUG [Listener at localhost.localdomain/33637] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-05-22 03:53:02,852 INFO [Listener at localhost.localdomain/33637] ipc.NettyRpcServer(120): Bind to /148.251.75.209:34247 2023-05-22 03:53:02,861 INFO [Listener at localhost.localdomain/33637] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:53:02,865 INFO [Listener at localhost.localdomain/33637] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:53:02,882 INFO [Listener at localhost.localdomain/33637] zookeeper.RecoverableZooKeeper(93): Process identifier=master:34247 connecting to ZooKeeper ensemble=127.0.0.1:58845 2023-05-22 03:53:02,914 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:342470x0, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-05-22 03:53:02,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:34247-0x101714ce2ce0000 connected 2023-05-22 03:53:02,936 DEBUG [Listener at localhost.localdomain/33637] zookeeper.ZKUtil(164): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:53:02,937 DEBUG [Listener at localhost.localdomain/33637] zookeeper.ZKUtil(164): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:53:02,941 DEBUG [Listener at localhost.localdomain/33637] zookeeper.ZKUtil(164): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-05-22 03:53:02,949 DEBUG [Listener at localhost.localdomain/33637] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34247 2023-05-22 03:53:02,949 DEBUG [Listener at localhost.localdomain/33637] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34247 2023-05-22 03:53:02,949 DEBUG [Listener at localhost.localdomain/33637] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34247 2023-05-22 03:53:02,950 DEBUG [Listener at localhost.localdomain/33637] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34247 2023-05-22 03:53:02,950 DEBUG [Listener at localhost.localdomain/33637] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34247 2023-05-22 03:53:02,954 INFO [Listener at localhost.localdomain/33637] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918, hbase.cluster.distributed=false 2023-05-22 03:53:03,009 INFO [Listener at localhost.localdomain/33637] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-05-22 03:53:03,009 INFO [Listener at localhost.localdomain/33637] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:53:03,009 INFO [Listener at localhost.localdomain/33637] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-05-22 03:53:03,010 INFO [Listener at localhost.localdomain/33637] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-05-22 03:53:03,010 INFO [Listener at localhost.localdomain/33637] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:53:03,010 INFO [Listener at localhost.localdomain/33637] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-05-22 03:53:03,014 INFO [Listener at localhost.localdomain/33637] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-05-22 03:53:03,016 INFO [Listener at localhost.localdomain/33637] ipc.NettyRpcServer(120): Bind to /148.251.75.209:37187 2023-05-22 03:53:03,018 INFO [Listener at localhost.localdomain/33637] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-05-22 03:53:03,023 DEBUG [Listener at localhost.localdomain/33637] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-05-22 03:53:03,024 INFO [Listener at localhost.localdomain/33637] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:53:03,026 INFO [Listener at localhost.localdomain/33637] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:53:03,027 INFO [Listener at localhost.localdomain/33637] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37187 connecting to ZooKeeper ensemble=127.0.0.1:58845 2023-05-22 03:53:03,031 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): regionserver:371870x0, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-05-22 03:53:03,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:37187-0x101714ce2ce0001 connected 2023-05-22 03:53:03,033 DEBUG [Listener at localhost.localdomain/33637] zookeeper.ZKUtil(164): regionserver:37187-0x101714ce2ce0001, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:53:03,034 DEBUG [Listener at localhost.localdomain/33637] zookeeper.ZKUtil(164): regionserver:37187-0x101714ce2ce0001, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:53:03,035 DEBUG [Listener at localhost.localdomain/33637] zookeeper.ZKUtil(164): regionserver:37187-0x101714ce2ce0001, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-05-22 03:53:03,035 DEBUG [Listener at localhost.localdomain/33637] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37187 2023-05-22 03:53:03,036 DEBUG [Listener at localhost.localdomain/33637] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37187 2023-05-22 03:53:03,036 DEBUG [Listener at localhost.localdomain/33637] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37187 2023-05-22 03:53:03,036 DEBUG [Listener at localhost.localdomain/33637] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37187 2023-05-22 03:53:03,037 DEBUG [Listener at localhost.localdomain/33637] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37187 2023-05-22 03:53:03,038 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,34247,1684727582098 2023-05-22 03:53:03,046 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-05-22 03:53:03,047 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,34247,1684727582098 2023-05-22 03:53:03,065 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-05-22 03:53:03,065 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): regionserver:37187-0x101714ce2ce0001, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-05-22 03:53:03,065 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:53:03,066 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-05-22 03:53:03,067 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,34247,1684727582098 from backup master directory 2023-05-22 03:53:03,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-05-22 03:53:03,069 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,34247,1684727582098 2023-05-22 03:53:03,069 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-05-22 03:53:03,070 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-05-22 03:53:03,070 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,34247,1684727582098 2023-05-22 03:53:03,072 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-05-22 03:53:03,073 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-05-22 03:53:03,152 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/hbase.id with ID: 820e8164-7b3d-49c7-ac93-decad15aeded 2023-05-22 03:53:03,196 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:53:03,334 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:53:03,382 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x487b983e to 127.0.0.1:58845 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:53:03,412 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b28cc22, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:53:03,431 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-05-22 03:53:03,433 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-05-22 03:53:03,442 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:53:03,469 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/MasterData/data/master/store-tmp 2023-05-22 03:53:03,496 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:53:03,497 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-05-22 03:53:03,497 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:53:03,497 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:53:03,497 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-05-22 03:53:03,497 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:53:03,497 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:53:03,497 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:53:03,499 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/MasterData/WALs/jenkins-hbase20.apache.org,34247,1684727582098 2023-05-22 03:53:03,518 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34247%2C1684727582098, suffix=, logDir=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/MasterData/WALs/jenkins-hbase20.apache.org,34247,1684727582098, archiveDir=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/MasterData/oldWALs, maxLogs=10 2023-05-22 03:53:03,535 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:307) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:70) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:881) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:574) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:515) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:160) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:295) at org.apache.hadoop.hbase.master.region.MasterRegion.createWAL(MasterRegion.java:200) at org.apache.hadoop.hbase.master.region.MasterRegion.bootstrap(MasterRegion.java:220) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:348) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:53:03,556 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/MasterData/WALs/jenkins-hbase20.apache.org,34247,1684727582098/jenkins-hbase20.apache.org%2C34247%2C1684727582098.1684727583534 2023-05-22 03:53:03,556 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:53:03,557 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:53:03,557 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:53:03,560 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:53:03,562 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:53:03,613 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:53:03,621 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-05-22 03:53:03,644 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-05-22 03:53:03,657 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:53:03,663 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:53:03,664 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:53:03,681 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:53:03,685 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:53:03,686 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=762624, jitterRate=-0.03027413785457611}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:53:03,686 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:53:03,688 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-05-22 03:53:03,704 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-05-22 03:53:03,704 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-05-22 03:53:03,706 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-05-22 03:53:03,708 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-05-22 03:53:03,736 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 28 msec 2023-05-22 03:53:03,737 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-05-22 03:53:03,759 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-05-22 03:53:03,764 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-05-22 03:53:03,790 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-05-22 03:53:03,793 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-22 03:53:03,795 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-05-22 03:53:03,800 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-05-22 03:53:03,803 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-05-22 03:53:03,806 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:53:03,807 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-05-22 03:53:03,808 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-05-22 03:53:03,819 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-05-22 03:53:03,834 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-05-22 03:53:03,834 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): regionserver:37187-0x101714ce2ce0001, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-05-22 03:53:03,835 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:53:03,835 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,34247,1684727582098, sessionid=0x101714ce2ce0000, setting cluster-up flag (Was=false) 2023-05-22 03:53:03,852 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:53:03,856 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-05-22 03:53:03,858 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,34247,1684727582098 2023-05-22 03:53:03,862 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:53:03,866 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-05-22 03:53:03,868 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,34247,1684727582098 2023-05-22 03:53:03,871 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/.hbase-snapshot/.tmp 2023-05-22 03:53:03,940 INFO [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(951): ClusterId : 820e8164-7b3d-49c7-ac93-decad15aeded 2023-05-22 03:53:03,944 DEBUG [RS:0;jenkins-hbase20:37187] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-05-22 03:53:03,949 DEBUG [RS:0;jenkins-hbase20:37187] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-05-22 03:53:03,949 DEBUG [RS:0;jenkins-hbase20:37187] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-05-22 03:53:03,951 DEBUG [RS:0;jenkins-hbase20:37187] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-05-22 03:53:03,952 DEBUG [RS:0;jenkins-hbase20:37187] zookeeper.ReadOnlyZKClient(139): Connect 0x6bd43af5 to 127.0.0.1:58845 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:53:03,956 DEBUG [RS:0;jenkins-hbase20:37187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6513c9ff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:53:03,957 DEBUG [RS:0;jenkins-hbase20:37187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@612c5185, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-05-22 03:53:03,971 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-05-22 03:53:03,980 DEBUG [RS:0;jenkins-hbase20:37187] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:37187 2023-05-22 03:53:03,981 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:53:03,981 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:53:03,982 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:53:03,982 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:53:03,982 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-05-22 03:53:03,982 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:53:03,982 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-05-22 03:53:03,982 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:53:03,984 INFO [RS:0;jenkins-hbase20:37187] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-05-22 03:53:03,984 INFO [RS:0;jenkins-hbase20:37187] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-05-22 03:53:03,984 DEBUG [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(1022): About to register with Master. 2023-05-22 03:53:03,985 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1684727613985 2023-05-22 03:53:03,988 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-05-22 03:53:03,989 INFO [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(2809): reportForDuty to master=jenkins-hbase20.apache.org,34247,1684727582098 with isa=jenkins-hbase20.apache.org/148.251.75.209:37187, startcode=1684727583008 2023-05-22 03:53:03,991 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-05-22 03:53:03,991 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-05-22 03:53:03,996 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-05-22 03:53:04,000 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-05-22 03:53:04,008 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-05-22 03:53:04,009 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-05-22 03:53:04,009 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-05-22 03:53:04,009 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-05-22 03:53:04,010 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-05-22 03:53:04,011 DEBUG [RS:0;jenkins-hbase20:37187] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-05-22 03:53:04,012 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-05-22 03:53:04,014 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-05-22 03:53:04,014 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-05-22 03:53:04,016 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-05-22 03:53:04,016 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-05-22 03:53:04,022 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727584018,5,FailOnTimeoutGroup] 2023-05-22 03:53:04,023 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727584023,5,FailOnTimeoutGroup] 2023-05-22 03:53:04,023 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-05-22 03:53:04,023 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-05-22 03:53:04,025 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-05-22 03:53:04,025 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-05-22 03:53:04,054 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-05-22 03:53:04,055 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-05-22 03:53:04,055 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918 2023-05-22 03:53:04,074 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:53:04,078 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-05-22 03:53:04,081 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/info 2023-05-22 03:53:04,082 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-05-22 03:53:04,083 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:53:04,083 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-05-22 03:53:04,087 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:53:04,088 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-05-22 03:53:04,090 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:53:04,090 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-05-22 03:53:04,092 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/table 2023-05-22 03:53:04,094 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-05-22 03:53:04,095 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:53:04,097 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740 2023-05-22 03:53:04,099 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740 2023-05-22 03:53:04,104 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-05-22 03:53:04,106 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-05-22 03:53:04,109 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:53:04,111 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=726921, jitterRate=-0.07567256689071655}}}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-05-22 03:53:04,111 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-05-22 03:53:04,111 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-05-22 03:53:04,111 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-05-22 03:53:04,111 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-05-22 03:53:04,111 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-05-22 03:53:04,111 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-05-22 03:53:04,112 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-05-22 03:53:04,112 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-05-22 03:53:04,116 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-05-22 03:53:04,117 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-05-22 03:53:04,123 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-05-22 03:53:04,135 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-05-22 03:53:04,141 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-05-22 03:53:04,160 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49575, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-05-22 03:53:04,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34247] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,37187,1684727583008 2023-05-22 03:53:04,185 DEBUG [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918 2023-05-22 03:53:04,185 DEBUG [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:36103 2023-05-22 03:53:04,185 DEBUG [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-05-22 03:53:04,190 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:53:04,190 DEBUG [RS:0;jenkins-hbase20:37187] zookeeper.ZKUtil(162): regionserver:37187-0x101714ce2ce0001, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,37187,1684727583008 2023-05-22 03:53:04,191 WARN [RS:0;jenkins-hbase20:37187] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-05-22 03:53:04,191 INFO [RS:0;jenkins-hbase20:37187] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:53:04,192 DEBUG [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(1946): logDir=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/WALs/jenkins-hbase20.apache.org,37187,1684727583008 2023-05-22 03:53:04,193 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,37187,1684727583008] 2023-05-22 03:53:04,202 DEBUG [RS:0;jenkins-hbase20:37187] zookeeper.ZKUtil(162): regionserver:37187-0x101714ce2ce0001, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,37187,1684727583008 2023-05-22 03:53:04,210 DEBUG [RS:0;jenkins-hbase20:37187] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-05-22 03:53:04,217 INFO [RS:0;jenkins-hbase20:37187] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-05-22 03:53:04,233 INFO [RS:0;jenkins-hbase20:37187] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-05-22 03:53:04,235 INFO [RS:0;jenkins-hbase20:37187] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-05-22 03:53:04,235 INFO [RS:0;jenkins-hbase20:37187] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:53:04,236 INFO [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-05-22 03:53:04,242 INFO [RS:0;jenkins-hbase20:37187] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-05-22 03:53:04,242 DEBUG [RS:0;jenkins-hbase20:37187] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:53:04,242 DEBUG [RS:0;jenkins-hbase20:37187] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:53:04,242 DEBUG [RS:0;jenkins-hbase20:37187] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:53:04,242 DEBUG [RS:0;jenkins-hbase20:37187] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:53:04,243 DEBUG [RS:0;jenkins-hbase20:37187] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:53:04,243 DEBUG [RS:0;jenkins-hbase20:37187] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-05-22 03:53:04,243 DEBUG [RS:0;jenkins-hbase20:37187] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:53:04,243 DEBUG [RS:0;jenkins-hbase20:37187] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:53:04,243 DEBUG [RS:0;jenkins-hbase20:37187] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:53:04,243 DEBUG [RS:0;jenkins-hbase20:37187] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:53:04,244 INFO [RS:0;jenkins-hbase20:37187] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-05-22 03:53:04,244 INFO [RS:0;jenkins-hbase20:37187] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-05-22 03:53:04,244 INFO [RS:0;jenkins-hbase20:37187] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-05-22 03:53:04,256 INFO [RS:0;jenkins-hbase20:37187] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-05-22 03:53:04,258 INFO [RS:0;jenkins-hbase20:37187] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37187,1684727583008-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:53:04,270 INFO [RS:0;jenkins-hbase20:37187] regionserver.Replication(203): jenkins-hbase20.apache.org,37187,1684727583008 started 2023-05-22 03:53:04,270 INFO [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,37187,1684727583008, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:37187, sessionid=0x101714ce2ce0001 2023-05-22 03:53:04,270 DEBUG [RS:0;jenkins-hbase20:37187] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-05-22 03:53:04,271 DEBUG [RS:0;jenkins-hbase20:37187] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,37187,1684727583008 2023-05-22 03:53:04,271 DEBUG [RS:0;jenkins-hbase20:37187] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,37187,1684727583008' 2023-05-22 03:53:04,271 DEBUG [RS:0;jenkins-hbase20:37187] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:53:04,271 DEBUG [RS:0;jenkins-hbase20:37187] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:53:04,272 DEBUG [RS:0;jenkins-hbase20:37187] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-05-22 03:53:04,272 DEBUG [RS:0;jenkins-hbase20:37187] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-05-22 03:53:04,272 DEBUG [RS:0;jenkins-hbase20:37187] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,37187,1684727583008 2023-05-22 03:53:04,272 DEBUG [RS:0;jenkins-hbase20:37187] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,37187,1684727583008' 2023-05-22 03:53:04,272 DEBUG [RS:0;jenkins-hbase20:37187] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-05-22 03:53:04,273 DEBUG [RS:0;jenkins-hbase20:37187] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-05-22 03:53:04,273 DEBUG [RS:0;jenkins-hbase20:37187] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-05-22 03:53:04,273 INFO [RS:0;jenkins-hbase20:37187] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-05-22 03:53:04,273 INFO [RS:0;jenkins-hbase20:37187] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-05-22 03:53:04,293 DEBUG [jenkins-hbase20:34247] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-05-22 03:53:04,296 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,37187,1684727583008, state=OPENING 2023-05-22 03:53:04,304 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-05-22 03:53:04,305 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:53:04,305 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-05-22 03:53:04,308 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,37187,1684727583008}] 2023-05-22 03:53:04,386 INFO [RS:0;jenkins-hbase20:37187] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C37187%2C1684727583008, suffix=, logDir=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/WALs/jenkins-hbase20.apache.org,37187,1684727583008, archiveDir=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/oldWALs, maxLogs=32 2023-05-22 03:53:04,401 INFO [RS:0;jenkins-hbase20:37187] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/WALs/jenkins-hbase20.apache.org,37187,1684727583008/jenkins-hbase20.apache.org%2C37187%2C1684727583008.1684727584390 2023-05-22 03:53:04,401 DEBUG [RS:0;jenkins-hbase20:37187] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:53:04,492 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,37187,1684727583008 2023-05-22 03:53:04,495 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-05-22 03:53:04,498 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:33138, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-05-22 03:53:04,510 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-05-22 03:53:04,511 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:53:04,514 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C37187%2C1684727583008.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/WALs/jenkins-hbase20.apache.org,37187,1684727583008, archiveDir=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/oldWALs, maxLogs=32 2023-05-22 03:53:04,530 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/WALs/jenkins-hbase20.apache.org,37187,1684727583008/jenkins-hbase20.apache.org%2C37187%2C1684727583008.meta.1684727584516.meta 2023-05-22 03:53:04,531 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK], DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK]] 2023-05-22 03:53:04,531 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:53:04,533 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-05-22 03:53:04,549 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-05-22 03:53:04,553 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-05-22 03:53:04,557 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-05-22 03:53:04,558 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:53:04,558 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-05-22 03:53:04,558 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-05-22 03:53:04,560 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-05-22 03:53:04,562 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/info 2023-05-22 03:53:04,562 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/info 2023-05-22 03:53:04,563 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-05-22 03:53:04,564 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:53:04,564 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-05-22 03:53:04,566 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:53:04,566 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:53:04,567 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-05-22 03:53:04,568 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:53:04,568 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-05-22 03:53:04,570 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/table 2023-05-22 03:53:04,570 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/table 2023-05-22 03:53:04,570 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-05-22 03:53:04,571 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:53:04,573 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740 2023-05-22 03:53:04,576 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740 2023-05-22 03:53:04,580 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-05-22 03:53:04,582 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-05-22 03:53:04,583 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=739194, jitterRate=-0.06006652116775513}}}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-05-22 03:53:04,584 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-05-22 03:53:04,593 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1684727584487 2023-05-22 03:53:04,609 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for hbase:meta,,1.1588230740 2023-05-22 03:53:04,610 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-05-22 03:53:04,611 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,37187,1684727583008, state=OPEN 2023-05-22 03:53:04,613 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-05-22 03:53:04,613 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-05-22 03:53:04,618 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-05-22 03:53:04,618 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,37187,1684727583008 in 305 msec 2023-05-22 03:53:04,624 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-05-22 03:53:04,625 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 496 msec 2023-05-22 03:53:04,632 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 714 msec 2023-05-22 03:53:04,633 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1684727584633, completionTime=-1 2023-05-22 03:53:04,633 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-05-22 03:53:04,633 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-05-22 03:53:04,687 DEBUG [hconnection-0x3125b5bc-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-05-22 03:53:04,690 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:33144, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-05-22 03:53:04,705 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-05-22 03:53:04,705 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1684727644705 2023-05-22 03:53:04,705 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1684727704705 2023-05-22 03:53:04,705 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 71 msec 2023-05-22 03:53:04,728 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34247,1684727582098-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:53:04,729 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34247,1684727582098-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:53:04,729 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34247,1684727582098-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:53:04,730 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:34247, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:53:04,730 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-05-22 03:53:04,736 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-05-22 03:53:04,742 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-05-22 03:53:04,744 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-05-22 03:53:04,753 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-05-22 03:53:04,755 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-05-22 03:53:04,759 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-05-22 03:53:04,778 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/.tmp/data/hbase/namespace/013ca0c9e5b3be714abfb1dd6c943061 2023-05-22 03:53:04,781 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/.tmp/data/hbase/namespace/013ca0c9e5b3be714abfb1dd6c943061 empty. 2023-05-22 03:53:04,781 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/.tmp/data/hbase/namespace/013ca0c9e5b3be714abfb1dd6c943061 2023-05-22 03:53:04,782 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-05-22 03:53:04,834 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-05-22 03:53:04,837 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 013ca0c9e5b3be714abfb1dd6c943061, NAME => 'hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/.tmp 2023-05-22 03:53:04,855 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:53:04,855 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 013ca0c9e5b3be714abfb1dd6c943061, disabling compactions & flushes 2023-05-22 03:53:04,855 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061. 2023-05-22 03:53:04,856 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061. 2023-05-22 03:53:04,856 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061. after waiting 0 ms 2023-05-22 03:53:04,856 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061. 2023-05-22 03:53:04,856 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061. 2023-05-22 03:53:04,856 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 013ca0c9e5b3be714abfb1dd6c943061: 2023-05-22 03:53:04,860 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-05-22 03:53:04,875 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727584863"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1684727584863"}]},"ts":"1684727584863"} 2023-05-22 03:53:04,897 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-05-22 03:53:04,899 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-05-22 03:53:04,903 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727584899"}]},"ts":"1684727584899"} 2023-05-22 03:53:04,907 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-05-22 03:53:04,916 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=013ca0c9e5b3be714abfb1dd6c943061, ASSIGN}] 2023-05-22 03:53:04,920 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=013ca0c9e5b3be714abfb1dd6c943061, ASSIGN 2023-05-22 03:53:04,922 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=013ca0c9e5b3be714abfb1dd6c943061, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,37187,1684727583008; forceNewPlan=false, retain=false 2023-05-22 03:53:05,073 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=013ca0c9e5b3be714abfb1dd6c943061, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,37187,1684727583008 2023-05-22 03:53:05,073 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727585073"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1684727585073"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1684727585073"}]},"ts":"1684727585073"} 2023-05-22 03:53:05,078 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 013ca0c9e5b3be714abfb1dd6c943061, server=jenkins-hbase20.apache.org,37187,1684727583008}] 2023-05-22 03:53:05,248 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061. 2023-05-22 03:53:05,251 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 013ca0c9e5b3be714abfb1dd6c943061, NAME => 'hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:53:05,253 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 013ca0c9e5b3be714abfb1dd6c943061 2023-05-22 03:53:05,253 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:53:05,254 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 013ca0c9e5b3be714abfb1dd6c943061 2023-05-22 03:53:05,254 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 013ca0c9e5b3be714abfb1dd6c943061 2023-05-22 03:53:05,256 INFO [StoreOpener-013ca0c9e5b3be714abfb1dd6c943061-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 013ca0c9e5b3be714abfb1dd6c943061 2023-05-22 03:53:05,260 DEBUG [StoreOpener-013ca0c9e5b3be714abfb1dd6c943061-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/namespace/013ca0c9e5b3be714abfb1dd6c943061/info 2023-05-22 03:53:05,260 DEBUG [StoreOpener-013ca0c9e5b3be714abfb1dd6c943061-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/namespace/013ca0c9e5b3be714abfb1dd6c943061/info 2023-05-22 03:53:05,261 INFO [StoreOpener-013ca0c9e5b3be714abfb1dd6c943061-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 013ca0c9e5b3be714abfb1dd6c943061 columnFamilyName info 2023-05-22 03:53:05,262 INFO [StoreOpener-013ca0c9e5b3be714abfb1dd6c943061-1] regionserver.HStore(310): Store=013ca0c9e5b3be714abfb1dd6c943061/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:53:05,265 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/namespace/013ca0c9e5b3be714abfb1dd6c943061 2023-05-22 03:53:05,266 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/namespace/013ca0c9e5b3be714abfb1dd6c943061 2023-05-22 03:53:05,275 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 013ca0c9e5b3be714abfb1dd6c943061 2023-05-22 03:53:05,280 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/namespace/013ca0c9e5b3be714abfb1dd6c943061/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:53:05,281 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 013ca0c9e5b3be714abfb1dd6c943061; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=876359, jitterRate=0.11434890329837799}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:53:05,281 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 013ca0c9e5b3be714abfb1dd6c943061: 2023-05-22 03:53:05,283 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061., pid=6, masterSystemTime=1684727585234 2023-05-22 03:53:05,287 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061. 2023-05-22 03:53:05,287 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061. 2023-05-22 03:53:05,288 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=013ca0c9e5b3be714abfb1dd6c943061, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,37187,1684727583008 2023-05-22 03:53:05,288 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727585287"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1684727585287"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1684727585287"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1684727585287"}]},"ts":"1684727585287"} 2023-05-22 03:53:05,297 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-05-22 03:53:05,298 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 013ca0c9e5b3be714abfb1dd6c943061, server=jenkins-hbase20.apache.org,37187,1684727583008 in 216 msec 2023-05-22 03:53:05,302 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-05-22 03:53:05,302 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=013ca0c9e5b3be714abfb1dd6c943061, ASSIGN in 381 msec 2023-05-22 03:53:05,304 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-05-22 03:53:05,305 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727585304"}]},"ts":"1684727585304"} 2023-05-22 03:53:05,308 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-05-22 03:53:05,311 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-05-22 03:53:05,314 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 566 msec 2023-05-22 03:53:05,356 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-05-22 03:53:05,358 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:53:05,358 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:53:05,400 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-05-22 03:53:05,417 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:53:05,423 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 30 msec 2023-05-22 03:53:05,435 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-05-22 03:53:05,450 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:53:05,454 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 19 msec 2023-05-22 03:53:05,474 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-05-22 03:53:05,477 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-05-22 03:53:05,477 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.407sec 2023-05-22 03:53:05,480 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-05-22 03:53:05,482 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-05-22 03:53:05,482 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-05-22 03:53:05,484 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34247,1684727582098-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-05-22 03:53:05,485 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34247,1684727582098-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-05-22 03:53:05,496 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-05-22 03:53:05,548 DEBUG [Listener at localhost.localdomain/33637] zookeeper.ReadOnlyZKClient(139): Connect 0x77dd2812 to 127.0.0.1:58845 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:53:05,552 DEBUG [Listener at localhost.localdomain/33637] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d20e4d6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:53:05,563 DEBUG [hconnection-0x8a57944-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-05-22 03:53:05,574 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:33158, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-05-22 03:53:05,583 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,34247,1684727582098 2023-05-22 03:53:05,584 INFO [Listener at localhost.localdomain/33637] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:53:05,593 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-05-22 03:53:05,593 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:53:05,594 INFO [Listener at localhost.localdomain/33637] master.MasterRpcServices(492): Client=null/null set balanceSwitch=false 2023-05-22 03:53:05,604 DEBUG [Listener at localhost.localdomain/33637] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-05-22 03:53:05,608 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53708, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-05-22 03:53:05,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34247] util.TableDescriptorChecker(340): MAX_FILESIZE for table descriptor or "hbase.hregion.max.filesize" (786432) is too small, which might cause over splitting into unmanageable number of regions. 2023-05-22 03:53:05,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34247] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (8192) is too small, which might cause very frequent flushing. 2023-05-22 03:53:05,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34247] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestLogRolling-testSlowSyncLogRolling', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-05-22 03:53:05,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34247] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestLogRolling-testSlowSyncLogRolling 2023-05-22 03:53:05,624 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestLogRolling-testSlowSyncLogRolling execute state=CREATE_TABLE_PRE_OPERATION 2023-05-22 03:53:05,626 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestLogRolling-testSlowSyncLogRolling execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-05-22 03:53:05,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34247] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestLogRolling-testSlowSyncLogRolling" procId is: 9 2023-05-22 03:53:05,630 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/.tmp/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6 2023-05-22 03:53:05,632 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/.tmp/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6 empty. 2023-05-22 03:53:05,634 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/.tmp/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6 2023-05-22 03:53:05,634 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestLogRolling-testSlowSyncLogRolling regions 2023-05-22 03:53:05,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34247] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-05-22 03:53:05,662 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/.tmp/data/default/TestLogRolling-testSlowSyncLogRolling/.tabledesc/.tableinfo.0000000001 2023-05-22 03:53:05,664 INFO [RegionOpenAndInit-TestLogRolling-testSlowSyncLogRolling-pool-0] regionserver.HRegion(7675): creating {ENCODED => 0c9372980f34569fe7a2686b5cdf61c6, NAME => 'TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestLogRolling-testSlowSyncLogRolling', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/.tmp 2023-05-22 03:53:05,678 DEBUG [RegionOpenAndInit-TestLogRolling-testSlowSyncLogRolling-pool-0] regionserver.HRegion(866): Instantiated TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:53:05,679 DEBUG [RegionOpenAndInit-TestLogRolling-testSlowSyncLogRolling-pool-0] regionserver.HRegion(1604): Closing 0c9372980f34569fe7a2686b5cdf61c6, disabling compactions & flushes 2023-05-22 03:53:05,679 INFO [RegionOpenAndInit-TestLogRolling-testSlowSyncLogRolling-pool-0] regionserver.HRegion(1626): Closing region TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6. 2023-05-22 03:53:05,679 DEBUG [RegionOpenAndInit-TestLogRolling-testSlowSyncLogRolling-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6. 2023-05-22 03:53:05,679 DEBUG [RegionOpenAndInit-TestLogRolling-testSlowSyncLogRolling-pool-0] regionserver.HRegion(1714): Acquired close lock on TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6. after waiting 0 ms 2023-05-22 03:53:05,679 DEBUG [RegionOpenAndInit-TestLogRolling-testSlowSyncLogRolling-pool-0] regionserver.HRegion(1724): Updates disabled for region TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6. 2023-05-22 03:53:05,679 INFO [RegionOpenAndInit-TestLogRolling-testSlowSyncLogRolling-pool-0] regionserver.HRegion(1838): Closed TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6. 2023-05-22 03:53:05,679 DEBUG [RegionOpenAndInit-TestLogRolling-testSlowSyncLogRolling-pool-0] regionserver.HRegion(1558): Region close journal for 0c9372980f34569fe7a2686b5cdf61c6: 2023-05-22 03:53:05,683 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestLogRolling-testSlowSyncLogRolling execute state=CREATE_TABLE_ADD_TO_META 2023-05-22 03:53:05,685 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1684727585684"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1684727585684"}]},"ts":"1684727585684"} 2023-05-22 03:53:05,688 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-05-22 03:53:05,689 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestLogRolling-testSlowSyncLogRolling execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-05-22 03:53:05,689 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestLogRolling-testSlowSyncLogRolling","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727585689"}]},"ts":"1684727585689"} 2023-05-22 03:53:05,692 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestLogRolling-testSlowSyncLogRolling, state=ENABLING in hbase:meta 2023-05-22 03:53:05,695 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestLogRolling-testSlowSyncLogRolling, region=0c9372980f34569fe7a2686b5cdf61c6, ASSIGN}] 2023-05-22 03:53:05,697 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestLogRolling-testSlowSyncLogRolling, region=0c9372980f34569fe7a2686b5cdf61c6, ASSIGN 2023-05-22 03:53:05,699 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestLogRolling-testSlowSyncLogRolling, region=0c9372980f34569fe7a2686b5cdf61c6, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,37187,1684727583008; forceNewPlan=false, retain=false 2023-05-22 03:53:05,851 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=0c9372980f34569fe7a2686b5cdf61c6, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,37187,1684727583008 2023-05-22 03:53:05,851 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1684727585850"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1684727585850"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1684727585850"}]},"ts":"1684727585850"} 2023-05-22 03:53:05,858 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 0c9372980f34569fe7a2686b5cdf61c6, server=jenkins-hbase20.apache.org,37187,1684727583008}] 2023-05-22 03:53:06,020 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6. 2023-05-22 03:53:06,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0c9372980f34569fe7a2686b5cdf61c6, NAME => 'TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:53:06,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestLogRolling-testSlowSyncLogRolling 0c9372980f34569fe7a2686b5cdf61c6 2023-05-22 03:53:06,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:53:06,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 0c9372980f34569fe7a2686b5cdf61c6 2023-05-22 03:53:06,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 0c9372980f34569fe7a2686b5cdf61c6 2023-05-22 03:53:06,023 INFO [StoreOpener-0c9372980f34569fe7a2686b5cdf61c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 0c9372980f34569fe7a2686b5cdf61c6 2023-05-22 03:53:06,026 DEBUG [StoreOpener-0c9372980f34569fe7a2686b5cdf61c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info 2023-05-22 03:53:06,026 DEBUG [StoreOpener-0c9372980f34569fe7a2686b5cdf61c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info 2023-05-22 03:53:06,027 INFO [StoreOpener-0c9372980f34569fe7a2686b5cdf61c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0c9372980f34569fe7a2686b5cdf61c6 columnFamilyName info 2023-05-22 03:53:06,028 INFO [StoreOpener-0c9372980f34569fe7a2686b5cdf61c6-1] regionserver.HStore(310): Store=0c9372980f34569fe7a2686b5cdf61c6/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:53:06,031 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6 2023-05-22 03:53:06,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6 2023-05-22 03:53:06,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 0c9372980f34569fe7a2686b5cdf61c6 2023-05-22 03:53:06,041 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:53:06,041 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 0c9372980f34569fe7a2686b5cdf61c6; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=689500, jitterRate=-0.12325647473335266}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:53:06,041 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 0c9372980f34569fe7a2686b5cdf61c6: 2023-05-22 03:53:06,043 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6., pid=11, masterSystemTime=1684727586014 2023-05-22 03:53:06,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6. 2023-05-22 03:53:06,045 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6. 2023-05-22 03:53:06,046 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=0c9372980f34569fe7a2686b5cdf61c6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,37187,1684727583008 2023-05-22 03:53:06,046 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1684727586046"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1684727586046"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1684727586046"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1684727586046"}]},"ts":"1684727586046"} 2023-05-22 03:53:06,053 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-05-22 03:53:06,053 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 0c9372980f34569fe7a2686b5cdf61c6, server=jenkins-hbase20.apache.org,37187,1684727583008 in 191 msec 2023-05-22 03:53:06,056 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-05-22 03:53:06,057 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestLogRolling-testSlowSyncLogRolling, region=0c9372980f34569fe7a2686b5cdf61c6, ASSIGN in 358 msec 2023-05-22 03:53:06,058 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestLogRolling-testSlowSyncLogRolling execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-05-22 03:53:06,058 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestLogRolling-testSlowSyncLogRolling","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727586058"}]},"ts":"1684727586058"} 2023-05-22 03:53:06,060 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestLogRolling-testSlowSyncLogRolling, state=ENABLED in hbase:meta 2023-05-22 03:53:06,063 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestLogRolling-testSlowSyncLogRolling execute state=CREATE_TABLE_POST_OPERATION 2023-05-22 03:53:06,066 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestLogRolling-testSlowSyncLogRolling in 443 msec 2023-05-22 03:53:10,077 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-22 03:53:10,216 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-05-22 03:53:10,219 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-05-22 03:53:10,220 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestLogRolling-testSlowSyncLogRolling' 2023-05-22 03:53:12,221 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-05-22 03:53:12,222 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-05-22 03:53:15,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34247] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-05-22 03:53:15,654 INFO [Listener at localhost.localdomain/33637] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestLogRolling-testSlowSyncLogRolling, procId: 9 completed 2023-05-22 03:53:15,662 DEBUG [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(2627): Found 1 regions for table TestLogRolling-testSlowSyncLogRolling 2023-05-22 03:53:15,664 DEBUG [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(2633): firstRegionName=TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6. 2023-05-22 03:53:27,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37187] regionserver.HRegion(9158): Flush requested on 0c9372980f34569fe7a2686b5cdf61c6 2023-05-22 03:53:27,720 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0c9372980f34569fe7a2686b5cdf61c6 1/1 column families, dataSize=7.36 KB heapSize=8.13 KB 2023-05-22 03:53:27,788 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=7.36 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/.tmp/info/083572b143ae412f813c802cc6fa4441 2023-05-22 03:53:27,833 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/.tmp/info/083572b143ae412f813c802cc6fa4441 as hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/083572b143ae412f813c802cc6fa4441 2023-05-22 03:53:27,842 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/083572b143ae412f813c802cc6fa4441, entries=7, sequenceid=11, filesize=12.1 K 2023-05-22 03:53:27,845 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~7.36 KB/7532, heapSize ~8.11 KB/8304, currentSize=0 B/0 for 0c9372980f34569fe7a2686b5cdf61c6 in 124ms, sequenceid=11, compaction requested=false 2023-05-22 03:53:27,846 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0c9372980f34569fe7a2686b5cdf61c6: 2023-05-22 03:53:35,944 INFO [sync.2] wal.AbstractFSWAL(1141): Slow sync cost: 203 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:53:38,152 INFO [sync.3] wal.AbstractFSWAL(1141): Slow sync cost: 202 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:53:40,358 INFO [sync.4] wal.AbstractFSWAL(1141): Slow sync cost: 200 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:53:42,562 INFO [sync.0] wal.AbstractFSWAL(1141): Slow sync cost: 201 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:53:42,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37187] regionserver.HRegion(9158): Flush requested on 0c9372980f34569fe7a2686b5cdf61c6 2023-05-22 03:53:42,563 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0c9372980f34569fe7a2686b5cdf61c6 1/1 column families, dataSize=7.36 KB heapSize=8.13 KB 2023-05-22 03:53:42,765 INFO [sync.1] wal.AbstractFSWAL(1141): Slow sync cost: 201 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:53:42,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=7.36 KB at sequenceid=21 (bloomFilter=true), to=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/.tmp/info/9c7528f7ed7c4c98903e08f474f7b795 2023-05-22 03:53:42,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/.tmp/info/9c7528f7ed7c4c98903e08f474f7b795 as hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/9c7528f7ed7c4c98903e08f474f7b795 2023-05-22 03:53:42,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/9c7528f7ed7c4c98903e08f474f7b795, entries=7, sequenceid=21, filesize=12.1 K 2023-05-22 03:53:43,014 INFO [sync.2] wal.AbstractFSWAL(1141): Slow sync cost: 202 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:53:43,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~7.36 KB/7532, heapSize ~8.11 KB/8304, currentSize=0 B/0 for 0c9372980f34569fe7a2686b5cdf61c6 in 452ms, sequenceid=21, compaction requested=false 2023-05-22 03:53:43,016 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0c9372980f34569fe7a2686b5cdf61c6: 2023-05-22 03:53:43,017 DEBUG [MemStoreFlusher.0] regionserver.ConstantSizeRegionSplitPolicy(109): Should split because info size=24.2 K, sizeToCheck=16.0 K 2023-05-22 03:53:43,017 DEBUG [MemStoreFlusher.0] regionserver.IncreasingToUpperBoundRegionSplitPolicy(84): regionsWithCommonTable=1 2023-05-22 03:53:43,020 DEBUG [MemStoreFlusher.0] regionserver.StoreUtils(129): cannot split hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/083572b143ae412f813c802cc6fa4441 because midkey is the same as first or last row 2023-05-22 03:53:44,769 INFO [sync.3] wal.AbstractFSWAL(1141): Slow sync cost: 202 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:53:46,973 WARN [sync.4] wal.AbstractFSWAL(1302): Requesting log roll because we exceeded slow sync threshold; count=7, threshold=5, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:53:46,974 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(197): WAL FSHLog jenkins-hbase20.apache.org%2C37187%2C1684727583008:(num 1684727584390) roll requested 2023-05-22 03:53:46,974 INFO [sync.4] wal.AbstractFSWAL(1141): Slow sync cost: 202 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:53:47,190 INFO [sync.0] wal.AbstractFSWAL(1141): Slow sync cost: 200 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:53:47,191 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/WALs/jenkins-hbase20.apache.org,37187,1684727583008/jenkins-hbase20.apache.org%2C37187%2C1684727583008.1684727584390 with entries=24, filesize=20.43 KB; new WAL /user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/WALs/jenkins-hbase20.apache.org,37187,1684727583008/jenkins-hbase20.apache.org%2C37187%2C1684727583008.1684727626975 2023-05-22 03:53:47,192 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:53:47,192 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/WALs/jenkins-hbase20.apache.org,37187,1684727583008/jenkins-hbase20.apache.org%2C37187%2C1684727583008.1684727584390 is not closed yet, will try archiving it next time 2023-05-22 03:53:56,993 INFO [Listener at localhost.localdomain/33637] hbase.Waiter(180): Waiting up to [10,000] milli-secs(wait.for.ratio=[1]) 2023-05-22 03:54:01,999 INFO [sync.0] wal.AbstractFSWAL(1141): Slow sync cost: 5002 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:54:01,999 WARN [sync.0] wal.AbstractFSWAL(1147): Requesting log roll because we exceeded slow sync threshold; time=5002 ms, threshold=5000 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:54:01,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37187] regionserver.HRegion(9158): Flush requested on 0c9372980f34569fe7a2686b5cdf61c6 2023-05-22 03:54:02,000 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(197): WAL FSHLog jenkins-hbase20.apache.org%2C37187%2C1684727583008:(num 1684727626975) roll requested 2023-05-22 03:54:02,000 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0c9372980f34569fe7a2686b5cdf61c6 1/1 column families, dataSize=7.36 KB heapSize=8.13 KB 2023-05-22 03:54:04,002 INFO [Listener at localhost.localdomain/33637] hbase.Waiter(180): Waiting up to [10,000] milli-secs(wait.for.ratio=[1]) 2023-05-22 03:54:07,003 INFO [sync.1] wal.AbstractFSWAL(1141): Slow sync cost: 5001 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:54:07,004 WARN [sync.1] wal.AbstractFSWAL(1147): Requesting log roll because we exceeded slow sync threshold; time=5001 ms, threshold=5000 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:54:07,021 INFO [sync.2] wal.AbstractFSWAL(1141): Slow sync cost: 5000 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:54:07,021 WARN [sync.2] wal.AbstractFSWAL(1147): Requesting log roll because we exceeded slow sync threshold; time=5000 ms, threshold=5000 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:54:07,023 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/WALs/jenkins-hbase20.apache.org,37187,1684727583008/jenkins-hbase20.apache.org%2C37187%2C1684727583008.1684727626975 with entries=6, filesize=6.07 KB; new WAL /user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/WALs/jenkins-hbase20.apache.org,37187,1684727583008/jenkins-hbase20.apache.org%2C37187%2C1684727583008.1684727642000 2023-05-22 03:54:07,024 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44901,DS-65aa84e3-834a-48ba-9786-b533e62f8b4b,DISK], DatanodeInfoWithStorage[127.0.0.1:39671,DS-149d08fb-a4de-4c4a-8397-01671b74caa8,DISK]] 2023-05-22 03:54:07,024 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/WALs/jenkins-hbase20.apache.org,37187,1684727583008/jenkins-hbase20.apache.org%2C37187%2C1684727583008.1684727626975 is not closed yet, will try archiving it next time 2023-05-22 03:54:07,029 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=7.36 KB at sequenceid=31 (bloomFilter=true), to=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/.tmp/info/96996bf25bb948d19cb5a458bd4875d0 2023-05-22 03:54:07,038 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/.tmp/info/96996bf25bb948d19cb5a458bd4875d0 as hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/96996bf25bb948d19cb5a458bd4875d0 2023-05-22 03:54:07,047 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/96996bf25bb948d19cb5a458bd4875d0, entries=7, sequenceid=31, filesize=12.1 K 2023-05-22 03:54:07,050 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~7.36 KB/7532, heapSize ~8.11 KB/8304, currentSize=0 B/0 for 0c9372980f34569fe7a2686b5cdf61c6 in 5050ms, sequenceid=31, compaction requested=true 2023-05-22 03:54:07,050 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0c9372980f34569fe7a2686b5cdf61c6: 2023-05-22 03:54:07,050 DEBUG [MemStoreFlusher.0] regionserver.ConstantSizeRegionSplitPolicy(109): Should split because info size=36.3 K, sizeToCheck=16.0 K 2023-05-22 03:54:07,050 DEBUG [MemStoreFlusher.0] regionserver.IncreasingToUpperBoundRegionSplitPolicy(84): regionsWithCommonTable=1 2023-05-22 03:54:07,050 DEBUG [MemStoreFlusher.0] regionserver.StoreUtils(129): cannot split hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/083572b143ae412f813c802cc6fa4441 because midkey is the same as first or last row 2023-05-22 03:54:07,052 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:54:07,052 DEBUG [RS:0;jenkins-hbase20:37187-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-05-22 03:54:07,057 DEBUG [RS:0;jenkins-hbase20:37187-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37197 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-05-22 03:54:07,059 DEBUG [RS:0;jenkins-hbase20:37187-shortCompactions-0] regionserver.HStore(1912): 0c9372980f34569fe7a2686b5cdf61c6/info is initiating minor compaction (all files) 2023-05-22 03:54:07,060 INFO [RS:0;jenkins-hbase20:37187-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0c9372980f34569fe7a2686b5cdf61c6/info in TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6. 2023-05-22 03:54:07,060 INFO [RS:0;jenkins-hbase20:37187-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/083572b143ae412f813c802cc6fa4441, hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/9c7528f7ed7c4c98903e08f474f7b795, hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/96996bf25bb948d19cb5a458bd4875d0] into tmpdir=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/.tmp, totalSize=36.3 K 2023-05-22 03:54:07,061 DEBUG [RS:0;jenkins-hbase20:37187-shortCompactions-0] compactions.Compactor(207): Compacting 083572b143ae412f813c802cc6fa4441, keycount=7, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1684727595671 2023-05-22 03:54:07,062 DEBUG [RS:0;jenkins-hbase20:37187-shortCompactions-0] compactions.Compactor(207): Compacting 9c7528f7ed7c4c98903e08f474f7b795, keycount=7, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=21, earliestPutTs=1684727609722 2023-05-22 03:54:07,063 DEBUG [RS:0;jenkins-hbase20:37187-shortCompactions-0] compactions.Compactor(207): Compacting 96996bf25bb948d19cb5a458bd4875d0, keycount=7, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=31, earliestPutTs=1684727624565 2023-05-22 03:54:07,090 INFO [RS:0;jenkins-hbase20:37187-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0c9372980f34569fe7a2686b5cdf61c6#info#compaction#3 average throughput is 10.77 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-05-22 03:54:07,111 DEBUG [RS:0;jenkins-hbase20:37187-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/.tmp/info/95bdd7635be847ba9216185681c92558 as hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/95bdd7635be847ba9216185681c92558 2023-05-22 03:54:07,130 INFO [RS:0;jenkins-hbase20:37187-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0c9372980f34569fe7a2686b5cdf61c6/info of 0c9372980f34569fe7a2686b5cdf61c6 into 95bdd7635be847ba9216185681c92558(size=27.0 K), total size for store is 27.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-05-22 03:54:07,130 DEBUG [RS:0;jenkins-hbase20:37187-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0c9372980f34569fe7a2686b5cdf61c6: 2023-05-22 03:54:07,130 INFO [RS:0;jenkins-hbase20:37187-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6., storeName=0c9372980f34569fe7a2686b5cdf61c6/info, priority=13, startTime=1684727647052; duration=0sec 2023-05-22 03:54:07,131 DEBUG [RS:0;jenkins-hbase20:37187-shortCompactions-0] regionserver.ConstantSizeRegionSplitPolicy(109): Should split because info size=27.0 K, sizeToCheck=16.0 K 2023-05-22 03:54:07,132 DEBUG [RS:0;jenkins-hbase20:37187-shortCompactions-0] regionserver.IncreasingToUpperBoundRegionSplitPolicy(84): regionsWithCommonTable=1 2023-05-22 03:54:07,132 DEBUG [RS:0;jenkins-hbase20:37187-shortCompactions-0] regionserver.StoreUtils(129): cannot split hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/95bdd7635be847ba9216185681c92558 because midkey is the same as first or last row 2023-05-22 03:54:07,132 DEBUG [RS:0;jenkins-hbase20:37187-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:54:19,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37187] regionserver.HRegion(9158): Flush requested on 0c9372980f34569fe7a2686b5cdf61c6 2023-05-22 03:54:19,135 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0c9372980f34569fe7a2686b5cdf61c6 1/1 column families, dataSize=7.36 KB heapSize=8.13 KB 2023-05-22 03:54:19,157 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=7.36 KB at sequenceid=42 (bloomFilter=true), to=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/.tmp/info/2da76530d7c4438eb67a535b2800c759 2023-05-22 03:54:19,166 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/.tmp/info/2da76530d7c4438eb67a535b2800c759 as hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/2da76530d7c4438eb67a535b2800c759 2023-05-22 03:54:19,175 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/2da76530d7c4438eb67a535b2800c759, entries=7, sequenceid=42, filesize=12.1 K 2023-05-22 03:54:19,176 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~7.36 KB/7532, heapSize ~8.11 KB/8304, currentSize=0 B/0 for 0c9372980f34569fe7a2686b5cdf61c6 in 41ms, sequenceid=42, compaction requested=false 2023-05-22 03:54:19,176 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0c9372980f34569fe7a2686b5cdf61c6: 2023-05-22 03:54:19,176 DEBUG [MemStoreFlusher.0] regionserver.ConstantSizeRegionSplitPolicy(109): Should split because info size=39.1 K, sizeToCheck=16.0 K 2023-05-22 03:54:19,176 DEBUG [MemStoreFlusher.0] regionserver.IncreasingToUpperBoundRegionSplitPolicy(84): regionsWithCommonTable=1 2023-05-22 03:54:19,177 DEBUG [MemStoreFlusher.0] regionserver.StoreUtils(129): cannot split hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/95bdd7635be847ba9216185681c92558 because midkey is the same as first or last row 2023-05-22 03:54:27,151 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-05-22 03:54:27,152 INFO [Listener at localhost.localdomain/33637] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-05-22 03:54:27,152 DEBUG [Listener at localhost.localdomain/33637] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77dd2812 to 127.0.0.1:58845 2023-05-22 03:54:27,152 DEBUG [Listener at localhost.localdomain/33637] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:54:27,153 DEBUG [Listener at localhost.localdomain/33637] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-05-22 03:54:27,153 DEBUG [Listener at localhost.localdomain/33637] util.JVMClusterUtil(257): Found active master hash=98625475, stopped=false 2023-05-22 03:54:27,153 INFO [Listener at localhost.localdomain/33637] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,34247,1684727582098 2023-05-22 03:54:27,154 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): regionserver:37187-0x101714ce2ce0001, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-05-22 03:54:27,154 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-05-22 03:54:27,154 INFO [Listener at localhost.localdomain/33637] procedure2.ProcedureExecutor(629): Stopping 2023-05-22 03:54:27,154 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:54:27,155 DEBUG [Listener at localhost.localdomain/33637] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x487b983e to 127.0.0.1:58845 2023-05-22 03:54:27,155 DEBUG [Listener at localhost.localdomain/33637] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:54:27,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37187-0x101714ce2ce0001, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:54:27,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:54:27,155 INFO [Listener at localhost.localdomain/33637] regionserver.HRegionServer(2295): ***** STOPPING region server 'jenkins-hbase20.apache.org,37187,1684727583008' ***** 2023-05-22 03:54:27,156 INFO [Listener at localhost.localdomain/33637] regionserver.HRegionServer(2309): STOPPED: Shutdown requested 2023-05-22 03:54:27,156 INFO [RS:0;jenkins-hbase20:37187] regionserver.HeapMemoryManager(220): Stopping 2023-05-22 03:54:27,156 INFO [RS:0;jenkins-hbase20:37187] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-05-22 03:54:27,156 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-05-22 03:54:27,156 INFO [RS:0;jenkins-hbase20:37187] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-05-22 03:54:27,156 INFO [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(3303): Received CLOSE for 013ca0c9e5b3be714abfb1dd6c943061 2023-05-22 03:54:27,157 INFO [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(3303): Received CLOSE for 0c9372980f34569fe7a2686b5cdf61c6 2023-05-22 03:54:27,157 INFO [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,37187,1684727583008 2023-05-22 03:54:27,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 013ca0c9e5b3be714abfb1dd6c943061, disabling compactions & flushes 2023-05-22 03:54:27,158 DEBUG [RS:0;jenkins-hbase20:37187] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6bd43af5 to 127.0.0.1:58845 2023-05-22 03:54:27,158 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061. 2023-05-22 03:54:27,158 DEBUG [RS:0;jenkins-hbase20:37187] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:54:27,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061. 2023-05-22 03:54:27,158 INFO [RS:0;jenkins-hbase20:37187] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-05-22 03:54:27,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061. after waiting 0 ms 2023-05-22 03:54:27,158 INFO [RS:0;jenkins-hbase20:37187] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-05-22 03:54:27,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061. 2023-05-22 03:54:27,158 INFO [RS:0;jenkins-hbase20:37187] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-05-22 03:54:27,158 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 013ca0c9e5b3be714abfb1dd6c943061 1/1 column families, dataSize=78 B heapSize=488 B 2023-05-22 03:54:27,158 INFO [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(3303): Received CLOSE for 1588230740 2023-05-22 03:54:27,159 INFO [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-05-22 03:54:27,159 DEBUG [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(1478): Online Regions={013ca0c9e5b3be714abfb1dd6c943061=hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061., 1588230740=hbase:meta,,1.1588230740, 0c9372980f34569fe7a2686b5cdf61c6=TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6.} 2023-05-22 03:54:27,159 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-05-22 03:54:27,159 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-05-22 03:54:27,159 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-05-22 03:54:27,159 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-05-22 03:54:27,160 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-05-22 03:54:27,160 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=2.87 KB heapSize=5.38 KB 2023-05-22 03:54:27,161 DEBUG [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(1504): Waiting on 013ca0c9e5b3be714abfb1dd6c943061, 0c9372980f34569fe7a2686b5cdf61c6, 1588230740 2023-05-22 03:54:27,183 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.64 KB at sequenceid=14 (bloomFilter=false), to=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/.tmp/info/f4a077dd3bf646069046de200c672862 2023-05-22 03:54:27,183 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/namespace/013ca0c9e5b3be714abfb1dd6c943061/.tmp/info/c30c3a55eb7b4ae2bdb97e8f99b8d537 2023-05-22 03:54:27,195 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/namespace/013ca0c9e5b3be714abfb1dd6c943061/.tmp/info/c30c3a55eb7b4ae2bdb97e8f99b8d537 as hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/namespace/013ca0c9e5b3be714abfb1dd6c943061/info/c30c3a55eb7b4ae2bdb97e8f99b8d537 2023-05-22 03:54:27,205 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/namespace/013ca0c9e5b3be714abfb1dd6c943061/info/c30c3a55eb7b4ae2bdb97e8f99b8d537, entries=2, sequenceid=6, filesize=4.8 K 2023-05-22 03:54:27,208 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 013ca0c9e5b3be714abfb1dd6c943061 in 50ms, sequenceid=6, compaction requested=false 2023-05-22 03:54:27,209 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=232 B at sequenceid=14 (bloomFilter=false), to=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/.tmp/table/d30ae76404d240a4a0a6c8ea064fccab 2023-05-22 03:54:27,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/namespace/013ca0c9e5b3be714abfb1dd6c943061/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-05-22 03:54:27,217 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061. 2023-05-22 03:54:27,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 013ca0c9e5b3be714abfb1dd6c943061: 2023-05-22 03:54:27,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1684727584743.013ca0c9e5b3be714abfb1dd6c943061. 2023-05-22 03:54:27,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 0c9372980f34569fe7a2686b5cdf61c6, disabling compactions & flushes 2023-05-22 03:54:27,217 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6. 2023-05-22 03:54:27,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6. 2023-05-22 03:54:27,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6. after waiting 0 ms 2023-05-22 03:54:27,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6. 2023-05-22 03:54:27,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 0c9372980f34569fe7a2686b5cdf61c6 1/1 column families, dataSize=3.15 KB heapSize=3.63 KB 2023-05-22 03:54:27,221 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/.tmp/info/f4a077dd3bf646069046de200c672862 as hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/info/f4a077dd3bf646069046de200c672862 2023-05-22 03:54:27,229 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/info/f4a077dd3bf646069046de200c672862, entries=20, sequenceid=14, filesize=7.4 K 2023-05-22 03:54:27,232 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/.tmp/table/d30ae76404d240a4a0a6c8ea064fccab as hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/table/d30ae76404d240a4a0a6c8ea064fccab 2023-05-22 03:54:27,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.15 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/.tmp/info/7a47802ae0a0484b9f48d56623c36450 2023-05-22 03:54:27,245 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/table/d30ae76404d240a4a0a6c8ea064fccab, entries=4, sequenceid=14, filesize=4.8 K 2023-05-22 03:54:27,246 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/.tmp/info/7a47802ae0a0484b9f48d56623c36450 as hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/7a47802ae0a0484b9f48d56623c36450 2023-05-22 03:54:27,246 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.87 KB/2938, heapSize ~5.09 KB/5216, currentSize=0 B/0 for 1588230740 in 86ms, sequenceid=14, compaction requested=false 2023-05-22 03:54:27,254 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/hbase/meta/1588230740/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=1 2023-05-22 03:54:27,255 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-05-22 03:54:27,256 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-05-22 03:54:27,256 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-05-22 03:54:27,257 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-05-22 03:54:27,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/7a47802ae0a0484b9f48d56623c36450, entries=3, sequenceid=48, filesize=7.9 K 2023-05-22 03:54:27,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.15 KB/3228, heapSize ~3.61 KB/3696, currentSize=0 B/0 for 0c9372980f34569fe7a2686b5cdf61c6 in 42ms, sequenceid=48, compaction requested=true 2023-05-22 03:54:27,262 DEBUG [StoreCloser-TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/083572b143ae412f813c802cc6fa4441, hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/9c7528f7ed7c4c98903e08f474f7b795, hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/96996bf25bb948d19cb5a458bd4875d0] to archive 2023-05-22 03:54:27,264 DEBUG [StoreCloser-TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-05-22 03:54:27,269 DEBUG [StoreCloser-TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/083572b143ae412f813c802cc6fa4441 to hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/archive/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/083572b143ae412f813c802cc6fa4441 2023-05-22 03:54:27,271 DEBUG [StoreCloser-TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/9c7528f7ed7c4c98903e08f474f7b795 to hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/archive/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/9c7528f7ed7c4c98903e08f474f7b795 2023-05-22 03:54:27,274 DEBUG [StoreCloser-TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/96996bf25bb948d19cb5a458bd4875d0 to hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/archive/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/info/96996bf25bb948d19cb5a458bd4875d0 2023-05-22 03:54:27,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/data/default/TestLogRolling-testSlowSyncLogRolling/0c9372980f34569fe7a2686b5cdf61c6/recovered.edits/51.seqid, newMaxSeqId=51, maxSeqId=1 2023-05-22 03:54:27,303 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6. 2023-05-22 03:54:27,303 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 0c9372980f34569fe7a2686b5cdf61c6: 2023-05-22 03:54:27,303 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed TestLogRolling-testSlowSyncLogRolling,,1684727585616.0c9372980f34569fe7a2686b5cdf61c6. 2023-05-22 03:54:27,318 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-05-22 03:54:27,318 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-05-22 03:54:27,361 INFO [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,37187,1684727583008; all regions closed. 2023-05-22 03:54:27,363 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/WALs/jenkins-hbase20.apache.org,37187,1684727583008 2023-05-22 03:54:27,376 DEBUG [RS:0;jenkins-hbase20:37187] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/oldWALs 2023-05-22 03:54:27,376 INFO [RS:0;jenkins-hbase20:37187] wal.AbstractFSWAL(1031): Closed WAL: FSHLog jenkins-hbase20.apache.org%2C37187%2C1684727583008.meta:.meta(num 1684727584516) 2023-05-22 03:54:27,377 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/WALs/jenkins-hbase20.apache.org,37187,1684727583008 2023-05-22 03:54:27,389 DEBUG [RS:0;jenkins-hbase20:37187] wal.AbstractFSWAL(1028): Moved 3 WAL file(s) to /user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/oldWALs 2023-05-22 03:54:27,389 INFO [RS:0;jenkins-hbase20:37187] wal.AbstractFSWAL(1031): Closed WAL: FSHLog jenkins-hbase20.apache.org%2C37187%2C1684727583008:(num 1684727642000) 2023-05-22 03:54:27,389 DEBUG [RS:0;jenkins-hbase20:37187] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:54:27,389 INFO [RS:0;jenkins-hbase20:37187] regionserver.LeaseManager(133): Closed leases 2023-05-22 03:54:27,389 INFO [RS:0;jenkins-hbase20:37187] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-05-22 03:54:27,390 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-05-22 03:54:27,390 INFO [RS:0;jenkins-hbase20:37187] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:37187 2023-05-22 03:54:27,396 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:54:27,396 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): regionserver:37187-0x101714ce2ce0001, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,37187,1684727583008 2023-05-22 03:54:27,396 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): regionserver:37187-0x101714ce2ce0001, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:54:27,396 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,37187,1684727583008] 2023-05-22 03:54:27,397 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,37187,1684727583008; numProcessing=1 2023-05-22 03:54:27,398 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,37187,1684727583008 already deleted, retry=false 2023-05-22 03:54:27,398 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,37187,1684727583008 expired; onlineServers=0 2023-05-22 03:54:27,398 INFO [RegionServerTracker-0] regionserver.HRegionServer(2295): ***** STOPPING region server 'jenkins-hbase20.apache.org,34247,1684727582098' ***** 2023-05-22 03:54:27,398 INFO [RegionServerTracker-0] regionserver.HRegionServer(2309): STOPPED: Cluster shutdown set; onlineServer=0 2023-05-22 03:54:27,398 DEBUG [M:0;jenkins-hbase20:34247] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44ddedf8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-05-22 03:54:27,398 INFO [M:0;jenkins-hbase20:34247] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,34247,1684727582098 2023-05-22 03:54:27,398 INFO [M:0;jenkins-hbase20:34247] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,34247,1684727582098; all regions closed. 2023-05-22 03:54:27,398 DEBUG [M:0;jenkins-hbase20:34247] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:54:27,399 DEBUG [M:0;jenkins-hbase20:34247] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-05-22 03:54:27,399 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-05-22 03:54:27,399 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727584018] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727584018,5,FailOnTimeoutGroup] 2023-05-22 03:54:27,399 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727584023] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727584023,5,FailOnTimeoutGroup] 2023-05-22 03:54:27,399 DEBUG [M:0;jenkins-hbase20:34247] cleaner.HFileCleaner(317): Stopping file delete threads 2023-05-22 03:54:27,401 INFO [M:0;jenkins-hbase20:34247] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-05-22 03:54:27,401 INFO [M:0;jenkins-hbase20:34247] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-05-22 03:54:27,401 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-05-22 03:54:27,401 INFO [M:0;jenkins-hbase20:34247] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-05-22 03:54:27,401 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:54:27,401 DEBUG [M:0;jenkins-hbase20:34247] master.HMaster(1512): Stopping service threads 2023-05-22 03:54:27,401 INFO [M:0;jenkins-hbase20:34247] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-05-22 03:54:27,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:54:27,402 INFO [M:0;jenkins-hbase20:34247] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-05-22 03:54:27,402 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-05-22 03:54:27,403 DEBUG [M:0;jenkins-hbase20:34247] zookeeper.ZKUtil(398): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-05-22 03:54:27,403 WARN [M:0;jenkins-hbase20:34247] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-05-22 03:54:27,403 INFO [M:0;jenkins-hbase20:34247] assignment.AssignmentManager(315): Stopping assignment manager 2023-05-22 03:54:27,403 INFO [M:0;jenkins-hbase20:34247] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-05-22 03:54:27,404 DEBUG [M:0;jenkins-hbase20:34247] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-05-22 03:54:27,404 INFO [M:0;jenkins-hbase20:34247] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:54:27,404 DEBUG [M:0;jenkins-hbase20:34247] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:54:27,404 DEBUG [M:0;jenkins-hbase20:34247] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-05-22 03:54:27,404 DEBUG [M:0;jenkins-hbase20:34247] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:54:27,404 INFO [M:0;jenkins-hbase20:34247] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=38.33 KB heapSize=46.76 KB 2023-05-22 03:54:27,421 INFO [M:0;jenkins-hbase20:34247] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.33 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/254e6eafacd040ba925e954eed1a28ab 2023-05-22 03:54:27,427 INFO [M:0;jenkins-hbase20:34247] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 254e6eafacd040ba925e954eed1a28ab 2023-05-22 03:54:27,429 DEBUG [M:0;jenkins-hbase20:34247] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/254e6eafacd040ba925e954eed1a28ab as hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/254e6eafacd040ba925e954eed1a28ab 2023-05-22 03:54:27,435 INFO [M:0;jenkins-hbase20:34247] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 254e6eafacd040ba925e954eed1a28ab 2023-05-22 03:54:27,435 INFO [M:0;jenkins-hbase20:34247] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/254e6eafacd040ba925e954eed1a28ab, entries=11, sequenceid=100, filesize=6.1 K 2023-05-22 03:54:27,436 INFO [M:0;jenkins-hbase20:34247] regionserver.HRegion(2948): Finished flush of dataSize ~38.33 KB/39246, heapSize ~46.74 KB/47864, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 32ms, sequenceid=100, compaction requested=false 2023-05-22 03:54:27,437 INFO [M:0;jenkins-hbase20:34247] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:54:27,437 DEBUG [M:0;jenkins-hbase20:34247] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:54:27,438 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/MasterData/WALs/jenkins-hbase20.apache.org,34247,1684727582098 2023-05-22 03:54:27,441 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-05-22 03:54:27,442 INFO [M:0;jenkins-hbase20:34247] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-05-22 03:54:27,442 INFO [M:0;jenkins-hbase20:34247] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:34247 2023-05-22 03:54:27,444 DEBUG [M:0;jenkins-hbase20:34247] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,34247,1684727582098 already deleted, retry=false 2023-05-22 03:54:27,497 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): regionserver:37187-0x101714ce2ce0001, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:54:27,497 INFO [RS:0;jenkins-hbase20:37187] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,37187,1684727583008; zookeeper connection closed. 2023-05-22 03:54:27,497 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): regionserver:37187-0x101714ce2ce0001, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:54:27,498 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@15d2bf30] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@15d2bf30 2023-05-22 03:54:27,499 INFO [Listener at localhost.localdomain/33637] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-05-22 03:54:27,598 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:54:27,598 INFO [M:0;jenkins-hbase20:34247] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,34247,1684727582098; zookeeper connection closed. 2023-05-22 03:54:27,598 DEBUG [Listener at localhost.localdomain/33637-EventThread] zookeeper.ZKWatcher(600): master:34247-0x101714ce2ce0000, quorum=127.0.0.1:58845, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:54:27,602 WARN [Listener at localhost.localdomain/33637] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:54:27,609 INFO [Listener at localhost.localdomain/33637] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:54:27,715 WARN [BP-192059923-148.251.75.209-1684727579365 heartbeating to localhost.localdomain/127.0.0.1:36103] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:54:27,715 WARN [BP-192059923-148.251.75.209-1684727579365 heartbeating to localhost.localdomain/127.0.0.1:36103] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-192059923-148.251.75.209-1684727579365 (Datanode Uuid 180eb67b-bdae-4f32-a969-3e7d24f5e7ac) service to localhost.localdomain/127.0.0.1:36103 2023-05-22 03:54:27,717 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/cluster_ffd434c3-71a0-71a8-8706-dca49e1cb782/dfs/data/data3/current/BP-192059923-148.251.75.209-1684727579365] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:54:27,717 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/cluster_ffd434c3-71a0-71a8-8706-dca49e1cb782/dfs/data/data4/current/BP-192059923-148.251.75.209-1684727579365] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:54:27,718 WARN [Listener at localhost.localdomain/33637] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:54:27,721 INFO [Listener at localhost.localdomain/33637] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:54:27,829 WARN [BP-192059923-148.251.75.209-1684727579365 heartbeating to localhost.localdomain/127.0.0.1:36103] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:54:27,829 WARN [BP-192059923-148.251.75.209-1684727579365 heartbeating to localhost.localdomain/127.0.0.1:36103] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-192059923-148.251.75.209-1684727579365 (Datanode Uuid 9f31610a-e069-4fa6-854b-5af1903b9198) service to localhost.localdomain/127.0.0.1:36103 2023-05-22 03:54:27,831 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/cluster_ffd434c3-71a0-71a8-8706-dca49e1cb782/dfs/data/data1/current/BP-192059923-148.251.75.209-1684727579365] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:54:27,831 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/cluster_ffd434c3-71a0-71a8-8706-dca49e1cb782/dfs/data/data2/current/BP-192059923-148.251.75.209-1684727579365] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:54:27,865 INFO [Listener at localhost.localdomain/33637] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-05-22 03:54:27,983 INFO [Listener at localhost.localdomain/33637] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-05-22 03:54:28,015 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-05-22 03:54:28,024 INFO [Listener at localhost.localdomain/33637] hbase.ResourceChecker(175): after: regionserver.wal.TestLogRolling#testSlowSyncLogRolling Thread=51 (was 10) Potentially hanging thread: nioEventLoopGroup-3-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcClient-timer-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost.localdomain/33637 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase20:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase20:0.leaseChecker java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.regionserver.LeaseManager.run(LeaseManager.java:82) Potentially hanging thread: nioEventLoopGroup-5-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (551965115) connection to localhost.localdomain/127.0.0.1:36103 from jenkins.hfs.0 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-3-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-1 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (551965115) connection to localhost.localdomain/127.0.0.1:36103 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: master/jenkins-hbase20:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.0@localhost.localdomain:36103 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3693) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase20:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: nioEventLoopGroup-2-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@4f29ccee java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:327) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Idle-Rpc-Conn-Sweeper-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SessionTracker java.lang.Thread.sleep(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost.localdomain:36103 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (551965115) connection to localhost.localdomain/127.0.0.1:36103 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-4-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=441 (was 264) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=135 (was 384), ProcessCount=170 (was 168) - ProcessCount LEAK? -, AvailableMemoryMB=7326 (was 8589) 2023-05-22 03:54:28,032 INFO [Listener at localhost.localdomain/33637] hbase.ResourceChecker(147): before: regionserver.wal.TestLogRolling#testLogRollOnDatanodeDeath Thread=52, OpenFileDescriptor=441, MaxFileDescriptor=60000, SystemLoadAverage=135, ProcessCount=170, AvailableMemoryMB=7326 2023-05-22 03:54:28,033 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-05-22 03:54:28,033 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/hadoop.log.dir so I do NOT create it in target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e 2023-05-22 03:54:28,033 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e340f10c-cc60-faad-2dcb-44a636787366/hadoop.tmp.dir so I do NOT create it in target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e 2023-05-22 03:54:28,033 INFO [Listener at localhost.localdomain/33637] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c, deleteOnExit=true 2023-05-22 03:54:28,033 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-05-22 03:54:28,033 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/test.cache.data in system properties and HBase conf 2023-05-22 03:54:28,033 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/hadoop.tmp.dir in system properties and HBase conf 2023-05-22 03:54:28,033 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/hadoop.log.dir in system properties and HBase conf 2023-05-22 03:54:28,034 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/mapreduce.cluster.local.dir in system properties and HBase conf 2023-05-22 03:54:28,034 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-05-22 03:54:28,034 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-05-22 03:54:28,034 DEBUG [Listener at localhost.localdomain/33637] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-05-22 03:54:28,034 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-05-22 03:54:28,034 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-05-22 03:54:28,034 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-05-22 03:54:28,035 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-05-22 03:54:28,035 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-05-22 03:54:28,035 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-05-22 03:54:28,035 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-05-22 03:54:28,035 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/dfs.journalnode.edits.dir in system properties and HBase conf 2023-05-22 03:54:28,035 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-05-22 03:54:28,035 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/nfs.dump.dir in system properties and HBase conf 2023-05-22 03:54:28,035 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/java.io.tmpdir in system properties and HBase conf 2023-05-22 03:54:28,035 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/dfs.journalnode.edits.dir in system properties and HBase conf 2023-05-22 03:54:28,036 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-05-22 03:54:28,036 INFO [Listener at localhost.localdomain/33637] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-05-22 03:54:28,037 WARN [Listener at localhost.localdomain/33637] blockmanagement.DatanodeManager(362): The given interval for marking stale datanode = 30000, which is larger than heartbeat expire interval 20000. 2023-05-22 03:54:28,038 WARN [Listener at localhost.localdomain/33637] conf.Configuration(1701): No unit for dfs.heartbeat.interval(1) assuming SECONDS 2023-05-22 03:54:28,039 WARN [Listener at localhost.localdomain/33637] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-05-22 03:54:28,066 WARN [Listener at localhost.localdomain/33637] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:54:28,068 INFO [Listener at localhost.localdomain/33637] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:54:28,073 INFO [Listener at localhost.localdomain/33637] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/java.io.tmpdir/Jetty_localhost_localdomain_41741_hdfs____agoy7l/webapp 2023-05-22 03:54:28,150 INFO [Listener at localhost.localdomain/33637] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:41741 2023-05-22 03:54:28,152 WARN [Listener at localhost.localdomain/33637] blockmanagement.DatanodeManager(362): The given interval for marking stale datanode = 30000, which is larger than heartbeat expire interval 20000. 2023-05-22 03:54:28,154 WARN [Listener at localhost.localdomain/33637] conf.Configuration(1701): No unit for dfs.heartbeat.interval(1) assuming SECONDS 2023-05-22 03:54:28,155 WARN [Listener at localhost.localdomain/33637] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-05-22 03:54:28,186 WARN [Listener at localhost.localdomain/37617] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:54:28,195 WARN [Listener at localhost.localdomain/37617] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:54:28,198 WARN [Listener at localhost.localdomain/37617] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:54:28,199 INFO [Listener at localhost.localdomain/37617] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:54:28,204 INFO [Listener at localhost.localdomain/37617] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/java.io.tmpdir/Jetty_localhost_46285_datanode____.7rw78s/webapp 2023-05-22 03:54:28,273 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-05-22 03:54:28,277 INFO [Listener at localhost.localdomain/37617] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46285 2023-05-22 03:54:28,283 WARN [Listener at localhost.localdomain/45871] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:54:28,296 WARN [Listener at localhost.localdomain/45871] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:54:28,299 WARN [Listener at localhost.localdomain/45871] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:54:28,301 INFO [Listener at localhost.localdomain/45871] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:54:28,306 INFO [Listener at localhost.localdomain/45871] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/java.io.tmpdir/Jetty_localhost_34057_datanode____n9wz08/webapp 2023-05-22 03:54:28,365 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x628a1d77aba9bbce: Processing first storage report for DS-c2b559d0-b47c-4ad6-a580-d9681724a793 from datanode b483ceca-c25c-402c-96f4-387a8926a181 2023-05-22 03:54:28,365 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x628a1d77aba9bbce: from storage DS-c2b559d0-b47c-4ad6-a580-d9681724a793 node DatanodeRegistration(127.0.0.1:37227, datanodeUuid=b483ceca-c25c-402c-96f4-387a8926a181, infoPort=32819, infoSecurePort=0, ipcPort=45871, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:54:28,365 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x628a1d77aba9bbce: Processing first storage report for DS-3a68265e-4c91-4166-b13b-fe7e8d877155 from datanode b483ceca-c25c-402c-96f4-387a8926a181 2023-05-22 03:54:28,365 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x628a1d77aba9bbce: from storage DS-3a68265e-4c91-4166-b13b-fe7e8d877155 node DatanodeRegistration(127.0.0.1:37227, datanodeUuid=b483ceca-c25c-402c-96f4-387a8926a181, infoPort=32819, infoSecurePort=0, ipcPort=45871, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:54:28,395 INFO [Listener at localhost.localdomain/45871] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34057 2023-05-22 03:54:28,402 WARN [Listener at localhost.localdomain/44663] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:54:28,526 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x91734987cf2596e6: Processing first storage report for DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46 from datanode a1e72017-c183-4eba-93c0-530451b9e3c0 2023-05-22 03:54:28,526 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x91734987cf2596e6: from storage DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46 node DatanodeRegistration(127.0.0.1:34581, datanodeUuid=a1e72017-c183-4eba-93c0-530451b9e3c0, infoPort=38125, infoSecurePort=0, ipcPort=44663, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:54:28,526 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x91734987cf2596e6: Processing first storage report for DS-3e0e64b5-429a-4785-9e0c-2d361afbcc63 from datanode a1e72017-c183-4eba-93c0-530451b9e3c0 2023-05-22 03:54:28,526 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x91734987cf2596e6: from storage DS-3e0e64b5-429a-4785-9e0c-2d361afbcc63 node DatanodeRegistration(127.0.0.1:34581, datanodeUuid=a1e72017-c183-4eba-93c0-530451b9e3c0, infoPort=38125, infoSecurePort=0, ipcPort=44663, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:54:28,616 DEBUG [Listener at localhost.localdomain/44663] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e 2023-05-22 03:54:28,620 INFO [Listener at localhost.localdomain/44663] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/zookeeper_0, clientPort=58195, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-05-22 03:54:28,621 INFO [Listener at localhost.localdomain/44663] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58195 2023-05-22 03:54:28,621 INFO [Listener at localhost.localdomain/44663] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:54:28,622 INFO [Listener at localhost.localdomain/44663] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:54:28,641 INFO [Listener at localhost.localdomain/44663] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91 with version=8 2023-05-22 03:54:28,641 INFO [Listener at localhost.localdomain/44663] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/hbase-staging 2023-05-22 03:54:28,643 INFO [Listener at localhost.localdomain/44663] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-05-22 03:54:28,643 INFO [Listener at localhost.localdomain/44663] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:54:28,644 INFO [Listener at localhost.localdomain/44663] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-05-22 03:54:28,644 INFO [Listener at localhost.localdomain/44663] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-05-22 03:54:28,644 INFO [Listener at localhost.localdomain/44663] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:54:28,644 INFO [Listener at localhost.localdomain/44663] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-05-22 03:54:28,644 INFO [Listener at localhost.localdomain/44663] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-05-22 03:54:28,645 INFO [Listener at localhost.localdomain/44663] ipc.NettyRpcServer(120): Bind to /148.251.75.209:46365 2023-05-22 03:54:28,646 INFO [Listener at localhost.localdomain/44663] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:54:28,647 INFO [Listener at localhost.localdomain/44663] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:54:28,649 INFO [Listener at localhost.localdomain/44663] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46365 connecting to ZooKeeper ensemble=127.0.0.1:58195 2023-05-22 03:54:28,654 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:463650x0, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-05-22 03:54:28,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:46365-0x101714e37b60000 connected 2023-05-22 03:54:28,670 DEBUG [Listener at localhost.localdomain/44663] zookeeper.ZKUtil(164): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:54:28,670 DEBUG [Listener at localhost.localdomain/44663] zookeeper.ZKUtil(164): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:54:28,671 DEBUG [Listener at localhost.localdomain/44663] zookeeper.ZKUtil(164): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-05-22 03:54:28,671 DEBUG [Listener at localhost.localdomain/44663] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46365 2023-05-22 03:54:28,671 DEBUG [Listener at localhost.localdomain/44663] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46365 2023-05-22 03:54:28,672 DEBUG [Listener at localhost.localdomain/44663] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46365 2023-05-22 03:54:28,672 DEBUG [Listener at localhost.localdomain/44663] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46365 2023-05-22 03:54:28,672 DEBUG [Listener at localhost.localdomain/44663] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46365 2023-05-22 03:54:28,672 INFO [Listener at localhost.localdomain/44663] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91, hbase.cluster.distributed=false 2023-05-22 03:54:28,683 INFO [Listener at localhost.localdomain/44663] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-05-22 03:54:28,683 INFO [Listener at localhost.localdomain/44663] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:54:28,683 INFO [Listener at localhost.localdomain/44663] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-05-22 03:54:28,683 INFO [Listener at localhost.localdomain/44663] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-05-22 03:54:28,683 INFO [Listener at localhost.localdomain/44663] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:54:28,684 INFO [Listener at localhost.localdomain/44663] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-05-22 03:54:28,684 INFO [Listener at localhost.localdomain/44663] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-05-22 03:54:28,685 INFO [Listener at localhost.localdomain/44663] ipc.NettyRpcServer(120): Bind to /148.251.75.209:45231 2023-05-22 03:54:28,685 INFO [Listener at localhost.localdomain/44663] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-05-22 03:54:28,686 DEBUG [Listener at localhost.localdomain/44663] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-05-22 03:54:28,686 INFO [Listener at localhost.localdomain/44663] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:54:28,687 INFO [Listener at localhost.localdomain/44663] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:54:28,688 INFO [Listener at localhost.localdomain/44663] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:45231 connecting to ZooKeeper ensemble=127.0.0.1:58195 2023-05-22 03:54:28,691 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): regionserver:452310x0, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-05-22 03:54:28,692 DEBUG [Listener at localhost.localdomain/44663] zookeeper.ZKUtil(164): regionserver:452310x0, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:54:28,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:45231-0x101714e37b60001 connected 2023-05-22 03:54:28,693 DEBUG [Listener at localhost.localdomain/44663] zookeeper.ZKUtil(164): regionserver:45231-0x101714e37b60001, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:54:28,693 DEBUG [Listener at localhost.localdomain/44663] zookeeper.ZKUtil(164): regionserver:45231-0x101714e37b60001, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-05-22 03:54:28,693 DEBUG [Listener at localhost.localdomain/44663] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45231 2023-05-22 03:54:28,694 DEBUG [Listener at localhost.localdomain/44663] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45231 2023-05-22 03:54:28,695 DEBUG [Listener at localhost.localdomain/44663] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45231 2023-05-22 03:54:28,695 DEBUG [Listener at localhost.localdomain/44663] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45231 2023-05-22 03:54:28,695 DEBUG [Listener at localhost.localdomain/44663] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45231 2023-05-22 03:54:28,696 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,46365,1684727668643 2023-05-22 03:54:28,697 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-05-22 03:54:28,697 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,46365,1684727668643 2023-05-22 03:54:28,698 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-05-22 03:54:28,698 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:54:28,698 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): regionserver:45231-0x101714e37b60001, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-05-22 03:54:28,700 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-05-22 03:54:28,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-05-22 03:54:28,700 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,46365,1684727668643 from backup master directory 2023-05-22 03:54:28,701 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,46365,1684727668643 2023-05-22 03:54:28,701 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-05-22 03:54:28,701 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-05-22 03:54:28,702 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,46365,1684727668643 2023-05-22 03:54:28,725 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/hbase.id with ID: 4a64b1ed-16c8-4e57-ad50-e9e98a311438 2023-05-22 03:54:28,738 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:54:28,741 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:54:28,751 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x64aeb685 to 127.0.0.1:58195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:54:28,757 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1bc9c7c7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:54:28,757 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-05-22 03:54:28,758 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-05-22 03:54:28,758 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:54:28,760 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/data/master/store-tmp 2023-05-22 03:54:28,772 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:54:28,773 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-05-22 03:54:28,773 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:54:28,773 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:54:28,773 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-05-22 03:54:28,773 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:54:28,773 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:54:28,773 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:54:28,774 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/WALs/jenkins-hbase20.apache.org,46365,1684727668643 2023-05-22 03:54:28,777 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C46365%2C1684727668643, suffix=, logDir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/WALs/jenkins-hbase20.apache.org,46365,1684727668643, archiveDir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/oldWALs, maxLogs=10 2023-05-22 03:54:28,784 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/WALs/jenkins-hbase20.apache.org,46365,1684727668643/jenkins-hbase20.apache.org%2C46365%2C1684727668643.1684727668777 2023-05-22 03:54:28,784 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34581,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK], DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] 2023-05-22 03:54:28,784 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:54:28,784 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:54:28,784 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:54:28,784 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:54:28,787 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:54:28,789 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-05-22 03:54:28,790 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-05-22 03:54:28,790 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:54:28,792 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:54:28,793 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:54:28,797 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:54:28,800 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:54:28,801 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=830780, jitterRate=0.056391745805740356}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:54:28,801 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:54:28,801 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-05-22 03:54:28,803 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-05-22 03:54:28,804 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-05-22 03:54:28,804 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-05-22 03:54:28,805 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-05-22 03:54:28,806 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-05-22 03:54:28,806 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-05-22 03:54:28,808 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-05-22 03:54:28,810 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-05-22 03:54:28,824 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-05-22 03:54:28,824 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-22 03:54:28,825 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-05-22 03:54:28,825 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-05-22 03:54:28,826 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-05-22 03:54:28,828 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:54:28,829 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-05-22 03:54:28,829 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-05-22 03:54:28,830 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-05-22 03:54:28,831 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-05-22 03:54:28,831 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): regionserver:45231-0x101714e37b60001, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-05-22 03:54:28,831 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:54:28,831 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,46365,1684727668643, sessionid=0x101714e37b60000, setting cluster-up flag (Was=false) 2023-05-22 03:54:28,836 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-05-22 03:54:28,837 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,46365,1684727668643 2023-05-22 03:54:28,839 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:54:28,842 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-05-22 03:54:28,843 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,46365,1684727668643 2023-05-22 03:54:28,844 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/.hbase-snapshot/.tmp 2023-05-22 03:54:28,847 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-05-22 03:54:28,847 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:54:28,847 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:54:28,847 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:54:28,847 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:54:28,847 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-05-22 03:54:28,847 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:28,847 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-05-22 03:54:28,847 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:28,852 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1684727698852 2023-05-22 03:54:28,853 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-05-22 03:54:28,853 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-05-22 03:54:28,853 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-05-22 03:54:28,853 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-05-22 03:54:28,853 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-05-22 03:54:28,853 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-05-22 03:54:28,853 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:28,854 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-05-22 03:54:28,855 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-05-22 03:54:28,855 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-05-22 03:54:28,855 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-05-22 03:54:28,855 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-05-22 03:54:28,855 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-05-22 03:54:28,855 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-05-22 03:54:28,856 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-05-22 03:54:28,858 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727668855,5,FailOnTimeoutGroup] 2023-05-22 03:54:28,862 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727668858,5,FailOnTimeoutGroup] 2023-05-22 03:54:28,862 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:28,863 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-05-22 03:54:28,863 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:28,863 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:28,879 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-05-22 03:54:28,880 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-05-22 03:54:28,880 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91 2023-05-22 03:54:28,893 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:54:28,895 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-05-22 03:54:28,897 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/meta/1588230740/info 2023-05-22 03:54:28,898 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-05-22 03:54:28,898 INFO [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(951): ClusterId : 4a64b1ed-16c8-4e57-ad50-e9e98a311438 2023-05-22 03:54:28,899 DEBUG [RS:0;jenkins-hbase20:45231] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-05-22 03:54:28,899 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:54:28,900 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-05-22 03:54:28,901 DEBUG [RS:0;jenkins-hbase20:45231] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-05-22 03:54:28,902 DEBUG [RS:0;jenkins-hbase20:45231] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-05-22 03:54:28,902 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:54:28,903 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-05-22 03:54:28,903 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:54:28,903 DEBUG [RS:0;jenkins-hbase20:45231] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-05-22 03:54:28,904 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-05-22 03:54:28,905 DEBUG [RS:0;jenkins-hbase20:45231] zookeeper.ReadOnlyZKClient(139): Connect 0x3c3de25d to 127.0.0.1:58195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:54:28,906 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/meta/1588230740/table 2023-05-22 03:54:28,907 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-05-22 03:54:28,907 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:54:28,913 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/meta/1588230740 2023-05-22 03:54:28,913 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/meta/1588230740 2023-05-22 03:54:28,915 DEBUG [RS:0;jenkins-hbase20:45231] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32fa1baa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:54:28,915 DEBUG [RS:0;jenkins-hbase20:45231] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36c5892d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-05-22 03:54:28,917 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-05-22 03:54:28,918 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-05-22 03:54:28,920 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:54:28,921 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=776898, jitterRate=-0.01212349534034729}}}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-05-22 03:54:28,921 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-05-22 03:54:28,921 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-05-22 03:54:28,921 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-05-22 03:54:28,921 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-05-22 03:54:28,921 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-05-22 03:54:28,922 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-05-22 03:54:28,922 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-05-22 03:54:28,922 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-05-22 03:54:28,924 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-05-22 03:54:28,924 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-05-22 03:54:28,924 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-05-22 03:54:28,926 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-05-22 03:54:28,928 DEBUG [RS:0;jenkins-hbase20:45231] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:45231 2023-05-22 03:54:28,928 INFO [RS:0;jenkins-hbase20:45231] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-05-22 03:54:28,928 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-05-22 03:54:28,928 INFO [RS:0;jenkins-hbase20:45231] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-05-22 03:54:28,928 DEBUG [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(1022): About to register with Master. 2023-05-22 03:54:28,928 INFO [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(2809): reportForDuty to master=jenkins-hbase20.apache.org,46365,1684727668643 with isa=jenkins-hbase20.apache.org/148.251.75.209:45231, startcode=1684727668683 2023-05-22 03:54:28,929 DEBUG [RS:0;jenkins-hbase20:45231] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-05-22 03:54:28,933 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41525, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-05-22 03:54:28,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46365] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,45231,1684727668683 2023-05-22 03:54:28,935 DEBUG [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91 2023-05-22 03:54:28,935 DEBUG [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:37617 2023-05-22 03:54:28,935 DEBUG [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-05-22 03:54:28,936 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:54:28,937 DEBUG [RS:0;jenkins-hbase20:45231] zookeeper.ZKUtil(162): regionserver:45231-0x101714e37b60001, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,45231,1684727668683 2023-05-22 03:54:28,937 WARN [RS:0;jenkins-hbase20:45231] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-05-22 03:54:28,937 INFO [RS:0;jenkins-hbase20:45231] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:54:28,937 DEBUG [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(1946): logDir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683 2023-05-22 03:54:28,937 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,45231,1684727668683] 2023-05-22 03:54:28,941 DEBUG [RS:0;jenkins-hbase20:45231] zookeeper.ZKUtil(162): regionserver:45231-0x101714e37b60001, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,45231,1684727668683 2023-05-22 03:54:28,942 DEBUG [RS:0;jenkins-hbase20:45231] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-05-22 03:54:28,943 INFO [RS:0;jenkins-hbase20:45231] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-05-22 03:54:28,945 INFO [RS:0;jenkins-hbase20:45231] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-05-22 03:54:28,945 INFO [RS:0;jenkins-hbase20:45231] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-05-22 03:54:28,945 INFO [RS:0;jenkins-hbase20:45231] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:28,946 INFO [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-05-22 03:54:28,947 INFO [RS:0;jenkins-hbase20:45231] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:28,947 DEBUG [RS:0;jenkins-hbase20:45231] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:28,947 DEBUG [RS:0;jenkins-hbase20:45231] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:28,947 DEBUG [RS:0;jenkins-hbase20:45231] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:28,947 DEBUG [RS:0;jenkins-hbase20:45231] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:28,948 DEBUG [RS:0;jenkins-hbase20:45231] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:28,948 DEBUG [RS:0;jenkins-hbase20:45231] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-05-22 03:54:28,948 DEBUG [RS:0;jenkins-hbase20:45231] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:28,948 DEBUG [RS:0;jenkins-hbase20:45231] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:28,948 DEBUG [RS:0;jenkins-hbase20:45231] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:28,948 DEBUG [RS:0;jenkins-hbase20:45231] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:28,949 INFO [RS:0;jenkins-hbase20:45231] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:28,949 INFO [RS:0;jenkins-hbase20:45231] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:28,949 INFO [RS:0;jenkins-hbase20:45231] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:28,961 INFO [RS:0;jenkins-hbase20:45231] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-05-22 03:54:28,961 INFO [RS:0;jenkins-hbase20:45231] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45231,1684727668683-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:28,973 INFO [RS:0;jenkins-hbase20:45231] regionserver.Replication(203): jenkins-hbase20.apache.org,45231,1684727668683 started 2023-05-22 03:54:28,974 INFO [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,45231,1684727668683, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:45231, sessionid=0x101714e37b60001 2023-05-22 03:54:28,974 DEBUG [RS:0;jenkins-hbase20:45231] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-05-22 03:54:28,974 DEBUG [RS:0;jenkins-hbase20:45231] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,45231,1684727668683 2023-05-22 03:54:28,974 DEBUG [RS:0;jenkins-hbase20:45231] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,45231,1684727668683' 2023-05-22 03:54:28,974 DEBUG [RS:0;jenkins-hbase20:45231] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:54:28,975 DEBUG [RS:0;jenkins-hbase20:45231] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:54:28,975 DEBUG [RS:0;jenkins-hbase20:45231] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-05-22 03:54:28,975 DEBUG [RS:0;jenkins-hbase20:45231] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-05-22 03:54:28,975 DEBUG [RS:0;jenkins-hbase20:45231] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,45231,1684727668683 2023-05-22 03:54:28,975 DEBUG [RS:0;jenkins-hbase20:45231] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,45231,1684727668683' 2023-05-22 03:54:28,975 DEBUG [RS:0;jenkins-hbase20:45231] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-05-22 03:54:28,976 DEBUG [RS:0;jenkins-hbase20:45231] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-05-22 03:54:28,977 DEBUG [RS:0;jenkins-hbase20:45231] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-05-22 03:54:28,977 INFO [RS:0;jenkins-hbase20:45231] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-05-22 03:54:28,977 INFO [RS:0;jenkins-hbase20:45231] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-05-22 03:54:29,078 DEBUG [jenkins-hbase20:46365] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-05-22 03:54:29,079 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,45231,1684727668683, state=OPENING 2023-05-22 03:54:29,080 INFO [RS:0;jenkins-hbase20:45231] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C45231%2C1684727668683, suffix=, logDir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683, archiveDir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/oldWALs, maxLogs=32 2023-05-22 03:54:29,080 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-05-22 03:54:29,081 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:54:29,081 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-05-22 03:54:29,081 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,45231,1684727668683}] 2023-05-22 03:54:29,094 INFO [RS:0;jenkins-hbase20:45231] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683/jenkins-hbase20.apache.org%2C45231%2C1684727668683.1684727669082 2023-05-22 03:54:29,094 DEBUG [RS:0;jenkins-hbase20:45231] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34581,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK], DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] 2023-05-22 03:54:29,239 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,45231,1684727668683 2023-05-22 03:54:29,239 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-05-22 03:54:29,243 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54460, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-05-22 03:54:29,251 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-05-22 03:54:29,252 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:54:29,255 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C45231%2C1684727668683.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683, archiveDir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/oldWALs, maxLogs=32 2023-05-22 03:54:29,268 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683/jenkins-hbase20.apache.org%2C45231%2C1684727668683.meta.1684727669257.meta 2023-05-22 03:54:29,268 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34581,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK], DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] 2023-05-22 03:54:29,268 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:54:29,269 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-05-22 03:54:29,269 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-05-22 03:54:29,270 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-05-22 03:54:29,270 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-05-22 03:54:29,270 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:54:29,270 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-05-22 03:54:29,270 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-05-22 03:54:29,272 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-05-22 03:54:29,274 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/meta/1588230740/info 2023-05-22 03:54:29,274 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/meta/1588230740/info 2023-05-22 03:54:29,275 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-05-22 03:54:29,275 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:54:29,275 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-05-22 03:54:29,277 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:54:29,277 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:54:29,277 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-05-22 03:54:29,278 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:54:29,278 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-05-22 03:54:29,279 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/meta/1588230740/table 2023-05-22 03:54:29,279 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/meta/1588230740/table 2023-05-22 03:54:29,281 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-05-22 03:54:29,281 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:54:29,283 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/meta/1588230740 2023-05-22 03:54:29,285 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/meta/1588230740 2023-05-22 03:54:29,287 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-05-22 03:54:29,290 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-05-22 03:54:29,291 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=756069, jitterRate=-0.03860945999622345}}}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-05-22 03:54:29,291 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-05-22 03:54:29,293 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1684727669239 2023-05-22 03:54:29,296 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for hbase:meta,,1.1588230740 2023-05-22 03:54:29,297 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-05-22 03:54:29,298 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,45231,1684727668683, state=OPEN 2023-05-22 03:54:29,300 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-05-22 03:54:29,300 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-05-22 03:54:29,304 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-05-22 03:54:29,304 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,45231,1684727668683 in 219 msec 2023-05-22 03:54:29,308 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-05-22 03:54:29,308 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 380 msec 2023-05-22 03:54:29,311 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 464 msec 2023-05-22 03:54:29,312 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1684727669311, completionTime=-1 2023-05-22 03:54:29,312 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-05-22 03:54:29,312 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-05-22 03:54:29,314 DEBUG [hconnection-0x6a9afa4f-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-05-22 03:54:29,316 INFO [RS-EventLoopGroup-6-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54464, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-05-22 03:54:29,318 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-05-22 03:54:29,318 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1684727729318 2023-05-22 03:54:29,318 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1684727789318 2023-05-22 03:54:29,318 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-05-22 03:54:29,324 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46365,1684727668643-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:29,324 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46365,1684727668643-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:29,324 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46365,1684727668643-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:29,324 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:46365, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:29,324 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:29,324 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-05-22 03:54:29,324 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-05-22 03:54:29,326 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-05-22 03:54:29,326 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-05-22 03:54:29,327 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-05-22 03:54:29,329 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-05-22 03:54:29,331 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/.tmp/data/hbase/namespace/d0d8f2fdf86d7a316e2ac72baf57c403 2023-05-22 03:54:29,331 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/.tmp/data/hbase/namespace/d0d8f2fdf86d7a316e2ac72baf57c403 empty. 2023-05-22 03:54:29,332 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/.tmp/data/hbase/namespace/d0d8f2fdf86d7a316e2ac72baf57c403 2023-05-22 03:54:29,332 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-05-22 03:54:29,348 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-05-22 03:54:29,349 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => d0d8f2fdf86d7a316e2ac72baf57c403, NAME => 'hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/.tmp 2023-05-22 03:54:29,365 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:54:29,366 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing d0d8f2fdf86d7a316e2ac72baf57c403, disabling compactions & flushes 2023-05-22 03:54:29,366 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. 2023-05-22 03:54:29,366 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. 2023-05-22 03:54:29,366 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. after waiting 0 ms 2023-05-22 03:54:29,366 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. 2023-05-22 03:54:29,366 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. 2023-05-22 03:54:29,366 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for d0d8f2fdf86d7a316e2ac72baf57c403: 2023-05-22 03:54:29,371 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-05-22 03:54:29,374 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727669373"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1684727669373"}]},"ts":"1684727669373"} 2023-05-22 03:54:29,376 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-05-22 03:54:29,378 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-05-22 03:54:29,378 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727669378"}]},"ts":"1684727669378"} 2023-05-22 03:54:29,380 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-05-22 03:54:29,383 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d0d8f2fdf86d7a316e2ac72baf57c403, ASSIGN}] 2023-05-22 03:54:29,386 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d0d8f2fdf86d7a316e2ac72baf57c403, ASSIGN 2023-05-22 03:54:29,387 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=d0d8f2fdf86d7a316e2ac72baf57c403, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,45231,1684727668683; forceNewPlan=false, retain=false 2023-05-22 03:54:29,539 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=d0d8f2fdf86d7a316e2ac72baf57c403, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,45231,1684727668683 2023-05-22 03:54:29,539 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727669538"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1684727669538"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1684727669538"}]},"ts":"1684727669538"} 2023-05-22 03:54:29,544 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure d0d8f2fdf86d7a316e2ac72baf57c403, server=jenkins-hbase20.apache.org,45231,1684727668683}] 2023-05-22 03:54:29,710 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. 2023-05-22 03:54:29,710 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d0d8f2fdf86d7a316e2ac72baf57c403, NAME => 'hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:54:29,710 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace d0d8f2fdf86d7a316e2ac72baf57c403 2023-05-22 03:54:29,711 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:54:29,711 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for d0d8f2fdf86d7a316e2ac72baf57c403 2023-05-22 03:54:29,711 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for d0d8f2fdf86d7a316e2ac72baf57c403 2023-05-22 03:54:29,712 INFO [StoreOpener-d0d8f2fdf86d7a316e2ac72baf57c403-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region d0d8f2fdf86d7a316e2ac72baf57c403 2023-05-22 03:54:29,714 DEBUG [StoreOpener-d0d8f2fdf86d7a316e2ac72baf57c403-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/namespace/d0d8f2fdf86d7a316e2ac72baf57c403/info 2023-05-22 03:54:29,714 DEBUG [StoreOpener-d0d8f2fdf86d7a316e2ac72baf57c403-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/namespace/d0d8f2fdf86d7a316e2ac72baf57c403/info 2023-05-22 03:54:29,715 INFO [StoreOpener-d0d8f2fdf86d7a316e2ac72baf57c403-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d0d8f2fdf86d7a316e2ac72baf57c403 columnFamilyName info 2023-05-22 03:54:29,715 INFO [StoreOpener-d0d8f2fdf86d7a316e2ac72baf57c403-1] regionserver.HStore(310): Store=d0d8f2fdf86d7a316e2ac72baf57c403/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:54:29,717 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/namespace/d0d8f2fdf86d7a316e2ac72baf57c403 2023-05-22 03:54:29,717 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/namespace/d0d8f2fdf86d7a316e2ac72baf57c403 2023-05-22 03:54:29,720 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for d0d8f2fdf86d7a316e2ac72baf57c403 2023-05-22 03:54:29,723 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/hbase/namespace/d0d8f2fdf86d7a316e2ac72baf57c403/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:54:29,723 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened d0d8f2fdf86d7a316e2ac72baf57c403; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=876620, jitterRate=0.11467999219894409}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:54:29,724 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for d0d8f2fdf86d7a316e2ac72baf57c403: 2023-05-22 03:54:29,725 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403., pid=6, masterSystemTime=1684727669700 2023-05-22 03:54:29,728 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. 2023-05-22 03:54:29,728 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. 2023-05-22 03:54:29,729 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=d0d8f2fdf86d7a316e2ac72baf57c403, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,45231,1684727668683 2023-05-22 03:54:29,729 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727669728"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1684727669728"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1684727669728"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1684727669728"}]},"ts":"1684727669728"} 2023-05-22 03:54:29,734 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-05-22 03:54:29,734 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure d0d8f2fdf86d7a316e2ac72baf57c403, server=jenkins-hbase20.apache.org,45231,1684727668683 in 187 msec 2023-05-22 03:54:29,736 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-05-22 03:54:29,737 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=d0d8f2fdf86d7a316e2ac72baf57c403, ASSIGN in 351 msec 2023-05-22 03:54:29,737 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-05-22 03:54:29,738 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727669738"}]},"ts":"1684727669738"} 2023-05-22 03:54:29,739 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-05-22 03:54:29,742 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-05-22 03:54:29,744 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 418 msec 2023-05-22 03:54:29,829 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-05-22 03:54:29,830 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:54:29,830 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:54:29,841 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-05-22 03:54:29,851 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:54:29,855 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 15 msec 2023-05-22 03:54:29,864 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-05-22 03:54:29,875 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:54:29,881 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-05-22 03:54:29,890 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-05-22 03:54:29,893 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-05-22 03:54:29,893 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 1.191sec 2023-05-22 03:54:29,893 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-05-22 03:54:29,893 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-05-22 03:54:29,893 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-05-22 03:54:29,893 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46365,1684727668643-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-05-22 03:54:29,893 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46365,1684727668643-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-05-22 03:54:29,897 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-05-22 03:54:29,900 DEBUG [Listener at localhost.localdomain/44663] zookeeper.ReadOnlyZKClient(139): Connect 0x268de280 to 127.0.0.1:58195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:54:29,907 DEBUG [Listener at localhost.localdomain/44663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@775f2953, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:54:29,909 DEBUG [hconnection-0x6837b560-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-05-22 03:54:29,911 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54466, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-05-22 03:54:29,913 INFO [Listener at localhost.localdomain/44663] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,46365,1684727668643 2023-05-22 03:54:29,914 INFO [Listener at localhost.localdomain/44663] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:54:29,917 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-05-22 03:54:29,917 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:54:29,919 INFO [Listener at localhost.localdomain/44663] master.MasterRpcServices(492): Client=null/null set balanceSwitch=false 2023-05-22 03:54:29,933 INFO [Listener at localhost.localdomain/44663] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-05-22 03:54:29,933 INFO [Listener at localhost.localdomain/44663] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:54:29,933 INFO [Listener at localhost.localdomain/44663] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-05-22 03:54:29,934 INFO [Listener at localhost.localdomain/44663] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-05-22 03:54:29,934 INFO [Listener at localhost.localdomain/44663] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:54:29,934 INFO [Listener at localhost.localdomain/44663] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-05-22 03:54:29,934 INFO [Listener at localhost.localdomain/44663] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-05-22 03:54:29,936 INFO [Listener at localhost.localdomain/44663] ipc.NettyRpcServer(120): Bind to /148.251.75.209:37737 2023-05-22 03:54:29,936 INFO [Listener at localhost.localdomain/44663] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-05-22 03:54:29,937 DEBUG [Listener at localhost.localdomain/44663] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-05-22 03:54:29,938 INFO [Listener at localhost.localdomain/44663] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:54:29,939 INFO [Listener at localhost.localdomain/44663] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:54:29,939 INFO [Listener at localhost.localdomain/44663] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37737 connecting to ZooKeeper ensemble=127.0.0.1:58195 2023-05-22 03:54:29,942 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): regionserver:377370x0, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-05-22 03:54:29,943 DEBUG [Listener at localhost.localdomain/44663] zookeeper.ZKUtil(162): regionserver:377370x0, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-05-22 03:54:29,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:37737-0x101714e37b60005 connected 2023-05-22 03:54:29,944 DEBUG [Listener at localhost.localdomain/44663] zookeeper.ZKUtil(162): regionserver:37737-0x101714e37b60005, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-05-22 03:54:29,945 DEBUG [Listener at localhost.localdomain/44663] zookeeper.ZKUtil(164): regionserver:37737-0x101714e37b60005, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-05-22 03:54:29,946 DEBUG [Listener at localhost.localdomain/44663] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37737 2023-05-22 03:54:29,946 DEBUG [Listener at localhost.localdomain/44663] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37737 2023-05-22 03:54:29,948 DEBUG [Listener at localhost.localdomain/44663] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37737 2023-05-22 03:54:29,948 DEBUG [Listener at localhost.localdomain/44663] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37737 2023-05-22 03:54:29,948 DEBUG [Listener at localhost.localdomain/44663] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37737 2023-05-22 03:54:29,950 INFO [RS:1;jenkins-hbase20:37737] regionserver.HRegionServer(951): ClusterId : 4a64b1ed-16c8-4e57-ad50-e9e98a311438 2023-05-22 03:54:29,951 DEBUG [RS:1;jenkins-hbase20:37737] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-05-22 03:54:29,957 DEBUG [RS:1;jenkins-hbase20:37737] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-05-22 03:54:29,957 DEBUG [RS:1;jenkins-hbase20:37737] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-05-22 03:54:29,959 DEBUG [RS:1;jenkins-hbase20:37737] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-05-22 03:54:29,960 DEBUG [RS:1;jenkins-hbase20:37737] zookeeper.ReadOnlyZKClient(139): Connect 0x4adee753 to 127.0.0.1:58195 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:54:29,963 DEBUG [RS:1;jenkins-hbase20:37737] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44190078, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:54:29,963 DEBUG [RS:1;jenkins-hbase20:37737] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40268c23, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-05-22 03:54:29,972 DEBUG [RS:1;jenkins-hbase20:37737] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase20:37737 2023-05-22 03:54:29,972 INFO [RS:1;jenkins-hbase20:37737] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-05-22 03:54:29,972 INFO [RS:1;jenkins-hbase20:37737] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-05-22 03:54:29,973 DEBUG [RS:1;jenkins-hbase20:37737] regionserver.HRegionServer(1022): About to register with Master. 2023-05-22 03:54:29,973 INFO [RS:1;jenkins-hbase20:37737] regionserver.HRegionServer(2809): reportForDuty to master=jenkins-hbase20.apache.org,46365,1684727668643 with isa=jenkins-hbase20.apache.org/148.251.75.209:37737, startcode=1684727669933 2023-05-22 03:54:29,973 DEBUG [RS:1;jenkins-hbase20:37737] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-05-22 03:54:29,977 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38687, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-05-22 03:54:29,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46365] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,37737,1684727669933 2023-05-22 03:54:29,978 DEBUG [RS:1;jenkins-hbase20:37737] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91 2023-05-22 03:54:29,978 DEBUG [RS:1;jenkins-hbase20:37737] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:37617 2023-05-22 03:54:29,978 DEBUG [RS:1;jenkins-hbase20:37737] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-05-22 03:54:29,979 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): regionserver:45231-0x101714e37b60001, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:54:29,979 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:54:29,980 DEBUG [RS:1;jenkins-hbase20:37737] zookeeper.ZKUtil(162): regionserver:37737-0x101714e37b60005, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,37737,1684727669933 2023-05-22 03:54:29,980 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,37737,1684727669933] 2023-05-22 03:54:29,980 WARN [RS:1;jenkins-hbase20:37737] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-05-22 03:54:29,980 INFO [RS:1;jenkins-hbase20:37737] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:54:29,980 DEBUG [RS:1;jenkins-hbase20:37737] regionserver.HRegionServer(1946): logDir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933 2023-05-22 03:54:29,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45231-0x101714e37b60001, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,45231,1684727668683 2023-05-22 03:54:29,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:45231-0x101714e37b60001, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,37737,1684727669933 2023-05-22 03:54:29,986 DEBUG [RS:1;jenkins-hbase20:37737] zookeeper.ZKUtil(162): regionserver:37737-0x101714e37b60005, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,45231,1684727668683 2023-05-22 03:54:29,986 DEBUG [RS:1;jenkins-hbase20:37737] zookeeper.ZKUtil(162): regionserver:37737-0x101714e37b60005, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,37737,1684727669933 2023-05-22 03:54:29,988 DEBUG [RS:1;jenkins-hbase20:37737] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-05-22 03:54:29,988 INFO [RS:1;jenkins-hbase20:37737] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-05-22 03:54:29,991 INFO [RS:1;jenkins-hbase20:37737] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-05-22 03:54:29,992 INFO [RS:1;jenkins-hbase20:37737] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-05-22 03:54:29,993 INFO [RS:1;jenkins-hbase20:37737] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:29,993 INFO [RS:1;jenkins-hbase20:37737] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-05-22 03:54:29,994 INFO [RS:1;jenkins-hbase20:37737] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:29,994 DEBUG [RS:1;jenkins-hbase20:37737] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:29,994 DEBUG [RS:1;jenkins-hbase20:37737] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:29,995 DEBUG [RS:1;jenkins-hbase20:37737] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:29,995 DEBUG [RS:1;jenkins-hbase20:37737] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:29,995 DEBUG [RS:1;jenkins-hbase20:37737] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:29,995 DEBUG [RS:1;jenkins-hbase20:37737] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-05-22 03:54:29,995 DEBUG [RS:1;jenkins-hbase20:37737] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:29,995 DEBUG [RS:1;jenkins-hbase20:37737] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:29,995 DEBUG [RS:1;jenkins-hbase20:37737] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:29,995 DEBUG [RS:1;jenkins-hbase20:37737] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:54:29,996 INFO [RS:1;jenkins-hbase20:37737] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:29,997 INFO [RS:1;jenkins-hbase20:37737] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:29,997 INFO [RS:1;jenkins-hbase20:37737] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:30,007 INFO [RS:1;jenkins-hbase20:37737] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-05-22 03:54:30,008 INFO [RS:1;jenkins-hbase20:37737] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37737,1684727669933-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:54:30,019 INFO [RS:1;jenkins-hbase20:37737] regionserver.Replication(203): jenkins-hbase20.apache.org,37737,1684727669933 started 2023-05-22 03:54:30,019 INFO [RS:1;jenkins-hbase20:37737] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,37737,1684727669933, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:37737, sessionid=0x101714e37b60005 2023-05-22 03:54:30,019 INFO [Listener at localhost.localdomain/44663] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:1;jenkins-hbase20:37737,5,FailOnTimeoutGroup] 2023-05-22 03:54:30,019 DEBUG [RS:1;jenkins-hbase20:37737] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-05-22 03:54:30,019 INFO [Listener at localhost.localdomain/44663] wal.TestLogRolling(323): Replication=2 2023-05-22 03:54:30,019 DEBUG [RS:1;jenkins-hbase20:37737] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,37737,1684727669933 2023-05-22 03:54:30,020 DEBUG [RS:1;jenkins-hbase20:37737] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,37737,1684727669933' 2023-05-22 03:54:30,020 DEBUG [RS:1;jenkins-hbase20:37737] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:54:30,021 DEBUG [RS:1;jenkins-hbase20:37737] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:54:30,022 DEBUG [Listener at localhost.localdomain/44663] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-05-22 03:54:30,022 DEBUG [RS:1;jenkins-hbase20:37737] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-05-22 03:54:30,022 DEBUG [RS:1;jenkins-hbase20:37737] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-05-22 03:54:30,022 DEBUG [RS:1;jenkins-hbase20:37737] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,37737,1684727669933 2023-05-22 03:54:30,023 DEBUG [RS:1;jenkins-hbase20:37737] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,37737,1684727669933' 2023-05-22 03:54:30,023 DEBUG [RS:1;jenkins-hbase20:37737] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-05-22 03:54:30,023 DEBUG [RS:1;jenkins-hbase20:37737] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-05-22 03:54:30,024 DEBUG [RS:1;jenkins-hbase20:37737] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-05-22 03:54:30,024 INFO [RS:1;jenkins-hbase20:37737] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-05-22 03:54:30,024 INFO [RS:1;jenkins-hbase20:37737] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-05-22 03:54:30,025 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48724, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-05-22 03:54:30,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46365] util.TableDescriptorChecker(340): MAX_FILESIZE for table descriptor or "hbase.hregion.max.filesize" (786432) is too small, which might cause over splitting into unmanageable number of regions. 2023-05-22 03:54:30,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46365] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (8192) is too small, which might cause very frequent flushing. 2023-05-22 03:54:30,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46365] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestLogRolling-testLogRollOnDatanodeDeath', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-05-22 03:54:30,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46365] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestLogRolling-testLogRollOnDatanodeDeath 2023-05-22 03:54:30,031 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestLogRolling-testLogRollOnDatanodeDeath execute state=CREATE_TABLE_PRE_OPERATION 2023-05-22 03:54:30,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46365] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestLogRolling-testLogRollOnDatanodeDeath" procId is: 9 2023-05-22 03:54:30,032 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestLogRolling-testLogRollOnDatanodeDeath execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-05-22 03:54:30,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46365] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-05-22 03:54:30,034 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/.tmp/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc 2023-05-22 03:54:30,035 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/.tmp/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc empty. 2023-05-22 03:54:30,035 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/.tmp/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc 2023-05-22 03:54:30,035 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestLogRolling-testLogRollOnDatanodeDeath regions 2023-05-22 03:54:30,052 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/.tmp/data/default/TestLogRolling-testLogRollOnDatanodeDeath/.tabledesc/.tableinfo.0000000001 2023-05-22 03:54:30,054 INFO [RegionOpenAndInit-TestLogRolling-testLogRollOnDatanodeDeath-pool-0] regionserver.HRegion(7675): creating {ENCODED => 65448373c7bf299ad9112f0ddcf80abc, NAME => 'TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestLogRolling-testLogRollOnDatanodeDeath', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/.tmp 2023-05-22 03:54:30,063 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRollOnDatanodeDeath-pool-0] regionserver.HRegion(866): Instantiated TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:54:30,064 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRollOnDatanodeDeath-pool-0] regionserver.HRegion(1604): Closing 65448373c7bf299ad9112f0ddcf80abc, disabling compactions & flushes 2023-05-22 03:54:30,064 INFO [RegionOpenAndInit-TestLogRolling-testLogRollOnDatanodeDeath-pool-0] regionserver.HRegion(1626): Closing region TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc. 2023-05-22 03:54:30,064 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRollOnDatanodeDeath-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc. 2023-05-22 03:54:30,064 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRollOnDatanodeDeath-pool-0] regionserver.HRegion(1714): Acquired close lock on TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc. after waiting 0 ms 2023-05-22 03:54:30,064 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRollOnDatanodeDeath-pool-0] regionserver.HRegion(1724): Updates disabled for region TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc. 2023-05-22 03:54:30,064 INFO [RegionOpenAndInit-TestLogRolling-testLogRollOnDatanodeDeath-pool-0] regionserver.HRegion(1838): Closed TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc. 2023-05-22 03:54:30,064 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRollOnDatanodeDeath-pool-0] regionserver.HRegion(1558): Region close journal for 65448373c7bf299ad9112f0ddcf80abc: 2023-05-22 03:54:30,067 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestLogRolling-testLogRollOnDatanodeDeath execute state=CREATE_TABLE_ADD_TO_META 2023-05-22 03:54:30,069 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1684727670069"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1684727670069"}]},"ts":"1684727670069"} 2023-05-22 03:54:30,071 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-05-22 03:54:30,072 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestLogRolling-testLogRollOnDatanodeDeath execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-05-22 03:54:30,072 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestLogRolling-testLogRollOnDatanodeDeath","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727670072"}]},"ts":"1684727670072"} 2023-05-22 03:54:30,074 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestLogRolling-testLogRollOnDatanodeDeath, state=ENABLING in hbase:meta 2023-05-22 03:54:30,080 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-05-22 03:54:30,082 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-22 03:54:30,082 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-05-22 03:54:30,082 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-05-22 03:54:30,083 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestLogRolling-testLogRollOnDatanodeDeath, region=65448373c7bf299ad9112f0ddcf80abc, ASSIGN}] 2023-05-22 03:54:30,085 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestLogRolling-testLogRollOnDatanodeDeath, region=65448373c7bf299ad9112f0ddcf80abc, ASSIGN 2023-05-22 03:54:30,086 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestLogRolling-testLogRollOnDatanodeDeath, region=65448373c7bf299ad9112f0ddcf80abc, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,37737,1684727669933; forceNewPlan=false, retain=false 2023-05-22 03:54:30,127 INFO [RS:1;jenkins-hbase20:37737] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C37737%2C1684727669933, suffix=, logDir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933, archiveDir=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/oldWALs, maxLogs=32 2023-05-22 03:54:30,145 INFO [RS:1;jenkins-hbase20:37737] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727670128 2023-05-22 03:54:30,146 DEBUG [RS:1;jenkins-hbase20:37737] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34581,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK], DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] 2023-05-22 03:54:30,242 INFO [jenkins-hbase20:46365] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-05-22 03:54:30,244 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=65448373c7bf299ad9112f0ddcf80abc, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,37737,1684727669933 2023-05-22 03:54:30,245 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1684727670244"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1684727670244"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1684727670244"}]},"ts":"1684727670244"} 2023-05-22 03:54:30,249 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 65448373c7bf299ad9112f0ddcf80abc, server=jenkins-hbase20.apache.org,37737,1684727669933}] 2023-05-22 03:54:30,404 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,37737,1684727669933 2023-05-22 03:54:30,404 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-05-22 03:54:30,407 INFO [RS-EventLoopGroup-7-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:45846, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-05-22 03:54:30,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc. 2023-05-22 03:54:30,413 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 65448373c7bf299ad9112f0ddcf80abc, NAME => 'TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:54:30,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestLogRolling-testLogRollOnDatanodeDeath 65448373c7bf299ad9112f0ddcf80abc 2023-05-22 03:54:30,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:54:30,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 65448373c7bf299ad9112f0ddcf80abc 2023-05-22 03:54:30,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 65448373c7bf299ad9112f0ddcf80abc 2023-05-22 03:54:30,415 INFO [StoreOpener-65448373c7bf299ad9112f0ddcf80abc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 65448373c7bf299ad9112f0ddcf80abc 2023-05-22 03:54:30,417 DEBUG [StoreOpener-65448373c7bf299ad9112f0ddcf80abc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/info 2023-05-22 03:54:30,417 DEBUG [StoreOpener-65448373c7bf299ad9112f0ddcf80abc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/info 2023-05-22 03:54:30,418 INFO [StoreOpener-65448373c7bf299ad9112f0ddcf80abc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 65448373c7bf299ad9112f0ddcf80abc columnFamilyName info 2023-05-22 03:54:30,418 INFO [StoreOpener-65448373c7bf299ad9112f0ddcf80abc-1] regionserver.HStore(310): Store=65448373c7bf299ad9112f0ddcf80abc/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:54:30,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc 2023-05-22 03:54:30,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc 2023-05-22 03:54:30,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 65448373c7bf299ad9112f0ddcf80abc 2023-05-22 03:54:30,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:54:30,431 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 65448373c7bf299ad9112f0ddcf80abc; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=778706, jitterRate=-0.009825125336647034}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:54:30,431 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 65448373c7bf299ad9112f0ddcf80abc: 2023-05-22 03:54:30,433 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc., pid=11, masterSystemTime=1684727670403 2023-05-22 03:54:30,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc. 2023-05-22 03:54:30,440 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc. 2023-05-22 03:54:30,441 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=65448373c7bf299ad9112f0ddcf80abc, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,37737,1684727669933 2023-05-22 03:54:30,441 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1684727670440"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1684727670440"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1684727670440"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1684727670440"}]},"ts":"1684727670440"} 2023-05-22 03:54:30,446 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-05-22 03:54:30,446 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 65448373c7bf299ad9112f0ddcf80abc, server=jenkins-hbase20.apache.org,37737,1684727669933 in 194 msec 2023-05-22 03:54:30,449 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-05-22 03:54:30,449 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestLogRolling-testLogRollOnDatanodeDeath, region=65448373c7bf299ad9112f0ddcf80abc, ASSIGN in 363 msec 2023-05-22 03:54:30,450 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestLogRolling-testLogRollOnDatanodeDeath execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-05-22 03:54:30,450 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestLogRolling-testLogRollOnDatanodeDeath","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727670450"}]},"ts":"1684727670450"} 2023-05-22 03:54:30,452 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestLogRolling-testLogRollOnDatanodeDeath, state=ENABLED in hbase:meta 2023-05-22 03:54:30,454 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestLogRolling-testLogRollOnDatanodeDeath execute state=CREATE_TABLE_POST_OPERATION 2023-05-22 03:54:30,457 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestLogRolling-testLogRollOnDatanodeDeath in 427 msec 2023-05-22 03:54:32,756 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-05-22 03:54:34,943 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-05-22 03:54:34,943 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-05-22 03:54:35,988 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestLogRolling-testLogRollOnDatanodeDeath' 2023-05-22 03:54:40,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46365] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-05-22 03:54:40,035 INFO [Listener at localhost.localdomain/44663] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestLogRolling-testLogRollOnDatanodeDeath, procId: 9 completed 2023-05-22 03:54:40,037 DEBUG [Listener at localhost.localdomain/44663] hbase.HBaseTestingUtility(2627): Found 1 regions for table TestLogRolling-testLogRollOnDatanodeDeath 2023-05-22 03:54:40,038 DEBUG [Listener at localhost.localdomain/44663] hbase.HBaseTestingUtility(2633): firstRegionName=TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc. 2023-05-22 03:54:40,057 WARN [Listener at localhost.localdomain/44663] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:54:40,060 WARN [Listener at localhost.localdomain/44663] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:54:40,061 INFO [Listener at localhost.localdomain/44663] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:54:40,067 INFO [Listener at localhost.localdomain/44663] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/java.io.tmpdir/Jetty_localhost_40631_datanode____.99j0u9/webapp 2023-05-22 03:54:40,147 INFO [Listener at localhost.localdomain/44663] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40631 2023-05-22 03:54:40,159 WARN [Listener at localhost.localdomain/35455] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:54:40,183 WARN [Listener at localhost.localdomain/35455] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:54:40,190 WARN [Listener at localhost.localdomain/35455] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:54:40,193 INFO [Listener at localhost.localdomain/35455] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:54:40,202 INFO [Listener at localhost.localdomain/35455] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/java.io.tmpdir/Jetty_localhost_39083_datanode____.om4o7o/webapp 2023-05-22 03:54:40,264 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5e2a3df961957042: Processing first storage report for DS-ecaf4275-1a8d-4716-8114-4b7b06871615 from datanode 418679ee-d7f2-47eb-a5d7-1782bb1228b4 2023-05-22 03:54:40,264 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5e2a3df961957042: from storage DS-ecaf4275-1a8d-4716-8114-4b7b06871615 node DatanodeRegistration(127.0.0.1:45105, datanodeUuid=418679ee-d7f2-47eb-a5d7-1782bb1228b4, infoPort=33161, infoSecurePort=0, ipcPort=35455, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:54:40,265 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5e2a3df961957042: Processing first storage report for DS-f6848728-ade2-4dff-9d15-66b5f49d4d6e from datanode 418679ee-d7f2-47eb-a5d7-1782bb1228b4 2023-05-22 03:54:40,265 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5e2a3df961957042: from storage DS-f6848728-ade2-4dff-9d15-66b5f49d4d6e node DatanodeRegistration(127.0.0.1:45105, datanodeUuid=418679ee-d7f2-47eb-a5d7-1782bb1228b4, infoPort=33161, infoSecurePort=0, ipcPort=35455, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:54:40,314 INFO [Listener at localhost.localdomain/35455] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39083 2023-05-22 03:54:40,324 WARN [Listener at localhost.localdomain/38377] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:54:40,353 WARN [Listener at localhost.localdomain/38377] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:54:40,355 WARN [Listener at localhost.localdomain/38377] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:54:40,356 INFO [Listener at localhost.localdomain/38377] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:54:40,360 INFO [Listener at localhost.localdomain/38377] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/java.io.tmpdir/Jetty_localhost_42583_datanode____.vk8qqv/webapp 2023-05-22 03:54:40,425 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x981d7a432a1e5f90: Processing first storage report for DS-75ff45d2-44e9-4a84-a4f0-954d0d3f85f0 from datanode 76a4a008-ec95-43bc-8f96-eb3b343c0901 2023-05-22 03:54:40,425 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x981d7a432a1e5f90: from storage DS-75ff45d2-44e9-4a84-a4f0-954d0d3f85f0 node DatanodeRegistration(127.0.0.1:44367, datanodeUuid=76a4a008-ec95-43bc-8f96-eb3b343c0901, infoPort=42075, infoSecurePort=0, ipcPort=38377, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:54:40,425 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x981d7a432a1e5f90: Processing first storage report for DS-9843a9c8-69a8-46b3-ba0f-5d8592a0aa52 from datanode 76a4a008-ec95-43bc-8f96-eb3b343c0901 2023-05-22 03:54:40,425 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x981d7a432a1e5f90: from storage DS-9843a9c8-69a8-46b3-ba0f-5d8592a0aa52 node DatanodeRegistration(127.0.0.1:44367, datanodeUuid=76a4a008-ec95-43bc-8f96-eb3b343c0901, infoPort=42075, infoSecurePort=0, ipcPort=38377, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:54:40,457 INFO [Listener at localhost.localdomain/38377] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42583 2023-05-22 03:54:40,466 WARN [Listener at localhost.localdomain/39439] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:54:40,538 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xad174368d441db57: Processing first storage report for DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc from datanode 11861ffc-e31f-4fbf-b895-d9945de0e9e8 2023-05-22 03:54:40,538 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xad174368d441db57: from storage DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc node DatanodeRegistration(127.0.0.1:44121, datanodeUuid=11861ffc-e31f-4fbf-b895-d9945de0e9e8, infoPort=39877, infoSecurePort=0, ipcPort=39439, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:54:40,538 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xad174368d441db57: Processing first storage report for DS-0a3e6579-ea25-445f-ab8e-6ce46687a181 from datanode 11861ffc-e31f-4fbf-b895-d9945de0e9e8 2023-05-22 03:54:40,538 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xad174368d441db57: from storage DS-0a3e6579-ea25-445f-ab8e-6ce46687a181 node DatanodeRegistration(127.0.0.1:44121, datanodeUuid=11861ffc-e31f-4fbf-b895-d9945de0e9e8, infoPort=39877, infoSecurePort=0, ipcPort=39439, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:54:40,574 WARN [Listener at localhost.localdomain/39439] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:54:40,578 WARN [ResponseProcessor for block BP-1513808452-148.251.75.209-1684727668040:blk_1073741829_1005] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-1513808452-148.251.75.209-1684727668040:blk_1073741829_1005 java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:456) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1080) 2023-05-22 03:54:40,578 WARN [ResponseProcessor for block BP-1513808452-148.251.75.209-1684727668040:blk_1073741838_1014] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-1513808452-148.251.75.209-1684727668040:blk_1073741838_1014 java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:456) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1080) 2023-05-22 03:54:40,579 WARN [DataStreamer for file /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/WALs/jenkins-hbase20.apache.org,46365,1684727668643/jenkins-hbase20.apache.org%2C46365%2C1684727668643.1684727668777 block BP-1513808452-148.251.75.209-1684727668040:blk_1073741829_1005] hdfs.DataStreamer(1548): Error Recovery for BP-1513808452-148.251.75.209-1684727668040:blk_1073741829_1005 in pipeline [DatanodeInfoWithStorage[127.0.0.1:34581,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK], DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]]: datanode 0(DatanodeInfoWithStorage[127.0.0.1:34581,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK]) is bad. 2023-05-22 03:54:40,579 WARN [DataStreamer for file /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727670128 block BP-1513808452-148.251.75.209-1684727668040:blk_1073741838_1014] hdfs.DataStreamer(1548): Error Recovery for BP-1513808452-148.251.75.209-1684727668040:blk_1073741838_1014 in pipeline [DatanodeInfoWithStorage[127.0.0.1:34581,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK], DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]]: datanode 0(DatanodeInfoWithStorage[127.0.0.1:34581,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK]) is bad. 2023-05-22 03:54:40,578 WARN [ResponseProcessor for block BP-1513808452-148.251.75.209-1684727668040:blk_1073741832_1008] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-1513808452-148.251.75.209-1684727668040:blk_1073741832_1008 java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:456) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1080) 2023-05-22 03:54:40,578 WARN [ResponseProcessor for block BP-1513808452-148.251.75.209-1684727668040:blk_1073741833_1009] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-1513808452-148.251.75.209-1684727668040:blk_1073741833_1009 java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:456) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1080) 2023-05-22 03:54:40,581 WARN [DataStreamer for file /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683/jenkins-hbase20.apache.org%2C45231%2C1684727668683.1684727669082 block BP-1513808452-148.251.75.209-1684727668040:blk_1073741832_1008] hdfs.DataStreamer(1548): Error Recovery for BP-1513808452-148.251.75.209-1684727668040:blk_1073741832_1008 in pipeline [DatanodeInfoWithStorage[127.0.0.1:34581,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK], DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]]: datanode 0(DatanodeInfoWithStorage[127.0.0.1:34581,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK]) is bad. 2023-05-22 03:54:40,632 WARN [DataStreamer for file /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683/jenkins-hbase20.apache.org%2C45231%2C1684727668683.meta.1684727669257.meta block BP-1513808452-148.251.75.209-1684727668040:blk_1073741833_1009] hdfs.DataStreamer(1548): Error Recovery for BP-1513808452-148.251.75.209-1684727668040:blk_1073741833_1009 in pipeline [DatanodeInfoWithStorage[127.0.0.1:34581,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK], DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]]: datanode 0(DatanodeInfoWithStorage[127.0.0.1:34581,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK]) is bad. 2023-05-22 03:54:40,638 INFO [Listener at localhost.localdomain/39439] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:54:40,649 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:46274 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741838_1014]] datanode.DataXceiver(323): 127.0.0.1:37227:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:46274 dst: /127.0.0.1:37227 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:37227 remote=/127.0.0.1:46274]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:40,650 WARN [PacketResponder: BP-1513808452-148.251.75.209-1684727668040:blk_1073741832_1008, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[127.0.0.1:37227]] datanode.BlockReceiver$PacketResponder(1486): IOException in BlockReceiver.run(): java.io.IOException: The stream is closed at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1630) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1565) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1478) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:40,650 WARN [PacketResponder: BP-1513808452-148.251.75.209-1684727668040:blk_1073741829_1005, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[127.0.0.1:37227]] datanode.BlockReceiver$PacketResponder(1486): IOException in BlockReceiver.run(): java.io.IOException: The stream is closed at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1630) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1565) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1478) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:40,650 WARN [PacketResponder: BP-1513808452-148.251.75.209-1684727668040:blk_1073741833_1009, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[127.0.0.1:37227]] datanode.BlockReceiver$PacketResponder(1486): IOException in BlockReceiver.run(): java.io.IOException: The stream is closed at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1630) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1565) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1478) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:40,650 WARN [PacketResponder: BP-1513808452-148.251.75.209-1684727668040:blk_1073741838_1014, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[127.0.0.1:37227]] datanode.BlockReceiver$PacketResponder(1486): IOException in BlockReceiver.run(): java.io.IOException: The stream is closed at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1630) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1565) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1478) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:40,649 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1810824824_17 at /127.0.0.1:46214 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741832_1008]] datanode.DataXceiver(323): 127.0.0.1:37227:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:46214 dst: /127.0.0.1:37227 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:37227 remote=/127.0.0.1:46214]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:40,649 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1237576890_17 at /127.0.0.1:46190 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741829_1005]] datanode.DataXceiver(323): 127.0.0.1:37227:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:46190 dst: /127.0.0.1:37227 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:37227 remote=/127.0.0.1:46190]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:40,649 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1810824824_17 at /127.0.0.1:46222 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741833_1009]] datanode.DataXceiver(323): 127.0.0.1:37227:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:46222 dst: /127.0.0.1:37227 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:37227 remote=/127.0.0.1:46222]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:40,652 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1810824824_17 at /127.0.0.1:58102 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741833_1009]] datanode.DataXceiver(323): 127.0.0.1:34581:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:58102 dst: /127.0.0.1:34581 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:40,652 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1237576890_17 at /127.0.0.1:58042 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741829_1005]] datanode.DataXceiver(323): 127.0.0.1:34581:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:58042 dst: /127.0.0.1:34581 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:40,651 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1810824824_17 at /127.0.0.1:58076 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741832_1008]] datanode.DataXceiver(323): 127.0.0.1:34581:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:58076 dst: /127.0.0.1:34581 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:40,656 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:58152 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741838_1014]] datanode.DataXceiver(323): 127.0.0.1:34581:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:58152 dst: /127.0.0.1:34581 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:40,742 WARN [BP-1513808452-148.251.75.209-1684727668040 heartbeating to localhost.localdomain/127.0.0.1:37617] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:54:40,742 WARN [BP-1513808452-148.251.75.209-1684727668040 heartbeating to localhost.localdomain/127.0.0.1:37617] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1513808452-148.251.75.209-1684727668040 (Datanode Uuid a1e72017-c183-4eba-93c0-530451b9e3c0) service to localhost.localdomain/127.0.0.1:37617 2023-05-22 03:54:40,743 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data3/current/BP-1513808452-148.251.75.209-1684727668040] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:54:40,743 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data4/current/BP-1513808452-148.251.75.209-1684727668040] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:54:40,745 WARN [Listener at localhost.localdomain/39439] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:54:40,746 WARN [ResponseProcessor for block BP-1513808452-148.251.75.209-1684727668040:blk_1073741838_1016] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-1513808452-148.251.75.209-1684727668040:blk_1073741838_1016 java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:456) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1080) 2023-05-22 03:54:40,746 WARN [ResponseProcessor for block BP-1513808452-148.251.75.209-1684727668040:blk_1073741832_1017] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-1513808452-148.251.75.209-1684727668040:blk_1073741832_1017 java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:456) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1080) 2023-05-22 03:54:40,746 WARN [ResponseProcessor for block BP-1513808452-148.251.75.209-1684727668040:blk_1073741829_1015] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-1513808452-148.251.75.209-1684727668040:blk_1073741829_1015 java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:456) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1080) 2023-05-22 03:54:40,746 WARN [ResponseProcessor for block BP-1513808452-148.251.75.209-1684727668040:blk_1073741833_1018] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-1513808452-148.251.75.209-1684727668040:blk_1073741833_1018 java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:456) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1080) 2023-05-22 03:54:40,751 INFO [Listener at localhost.localdomain/39439] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:54:40,857 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:56302 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741838_1014]] datanode.DataXceiver(323): 127.0.0.1:37227:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:56302 dst: /127.0.0.1:37227 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:40,859 WARN [BP-1513808452-148.251.75.209-1684727668040 heartbeating to localhost.localdomain/127.0.0.1:37617] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:54:40,858 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1810824824_17 at /127.0.0.1:56304 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741833_1009]] datanode.DataXceiver(323): 127.0.0.1:37227:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:56304 dst: /127.0.0.1:37227 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:40,858 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1810824824_17 at /127.0.0.1:56298 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741832_1008]] datanode.DataXceiver(323): 127.0.0.1:37227:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:56298 dst: /127.0.0.1:37227 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:40,857 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1237576890_17 at /127.0.0.1:56300 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741829_1005]] datanode.DataXceiver(323): 127.0.0.1:37227:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:56300 dst: /127.0.0.1:37227 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:40,860 WARN [BP-1513808452-148.251.75.209-1684727668040 heartbeating to localhost.localdomain/127.0.0.1:37617] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1513808452-148.251.75.209-1684727668040 (Datanode Uuid b483ceca-c25c-402c-96f4-387a8926a181) service to localhost.localdomain/127.0.0.1:37617 2023-05-22 03:54:40,864 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data1/current/BP-1513808452-148.251.75.209-1684727668040] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:54:40,865 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data2/current/BP-1513808452-148.251.75.209-1684727668040] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:54:40,871 DEBUG [Listener at localhost.localdomain/39439] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-05-22 03:54:40,874 INFO [RS-EventLoopGroup-7-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43748, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-05-22 03:54:40,875 WARN [RS:1;jenkins-hbase20:37737.append-pool-0] wal.FSHLog$RingBufferEventHandler(1203): Append sequenceId=4, requesting roll of WAL java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:54:40,876 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(197): WAL FSHLog jenkins-hbase20.apache.org%2C37737%2C1684727669933:(num 1684727670128) roll requested 2023-05-22 03:54:40,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37737] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=4, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1205) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1078) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:979) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.lang.Thread.run(Thread.java:750) Caused by: java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:54:40,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37737] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 1.2 K connection: 148.251.75.209:43748 deadline: 1684727690874, exception=org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=4, requesting roll of WAL 2023-05-22 03:54:40,891 WARN [regionserver/jenkins-hbase20:0.logRoller] wal.FSHLog(374): Failed sync-before-close but no outstanding appends; closing WALorg.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=4, requesting roll of WAL 2023-05-22 03:54:40,891 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727670128 with entries=1, filesize=467 B; new WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727680876 2023-05-22 03:54:40,893 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44367,DS-75ff45d2-44e9-4a84-a4f0-954d0d3f85f0,DISK], DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK]] 2023-05-22 03:54:40,893 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727670128 is not closed yet, will try archiving it next time 2023-05-22 03:54:40,893 WARN [Close-WAL-Writer-0] wal.AbstractProtobufLogWriter(237): Failed to write trailer, non-fatal, continuing... java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:54:40,893 WARN [Close-WAL-Writer-0] wal.FSHLog(466): Riding over failed WAL close of hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727670128; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:54:40,894 INFO [WAL-Archive-0] wal.AbstractFSWAL(783): Archiving hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727670128 to hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/oldWALs/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727670128 2023-05-22 03:54:52,958 INFO [Listener at localhost.localdomain/39439] wal.TestLogRolling(375): log.getCurrentFileName(): hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727680876 2023-05-22 03:54:52,958 WARN [Listener at localhost.localdomain/39439] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:54:52,960 WARN [ResponseProcessor for block BP-1513808452-148.251.75.209-1684727668040:blk_1073741839_1019] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-1513808452-148.251.75.209-1684727668040:blk_1073741839_1019 java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:456) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1080) 2023-05-22 03:54:52,960 WARN [DataStreamer for file /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727680876 block BP-1513808452-148.251.75.209-1684727668040:blk_1073741839_1019] hdfs.DataStreamer(1548): Error Recovery for BP-1513808452-148.251.75.209-1684727668040:blk_1073741839_1019 in pipeline [DatanodeInfoWithStorage[127.0.0.1:44367,DS-75ff45d2-44e9-4a84-a4f0-954d0d3f85f0,DISK], DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK]]: datanode 0(DatanodeInfoWithStorage[127.0.0.1:44367,DS-75ff45d2-44e9-4a84-a4f0-954d0d3f85f0,DISK]) is bad. 2023-05-22 03:54:52,967 INFO [Listener at localhost.localdomain/39439] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:54:52,969 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:50124 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741839_1019]] datanode.DataXceiver(323): 127.0.0.1:44121:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:50124 dst: /127.0.0.1:44121 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:44121 remote=/127.0.0.1:50124]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:52,970 WARN [PacketResponder: BP-1513808452-148.251.75.209-1684727668040:blk_1073741839_1019, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[127.0.0.1:44121]] datanode.BlockReceiver$PacketResponder(1486): IOException in BlockReceiver.run(): java.io.IOException: The stream is closed at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1630) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1565) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1478) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:52,971 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:55696 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741839_1019]] datanode.DataXceiver(323): 127.0.0.1:44367:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:55696 dst: /127.0.0.1:44367 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:53,073 WARN [BP-1513808452-148.251.75.209-1684727668040 heartbeating to localhost.localdomain/127.0.0.1:37617] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:54:53,073 WARN [BP-1513808452-148.251.75.209-1684727668040 heartbeating to localhost.localdomain/127.0.0.1:37617] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1513808452-148.251.75.209-1684727668040 (Datanode Uuid 76a4a008-ec95-43bc-8f96-eb3b343c0901) service to localhost.localdomain/127.0.0.1:37617 2023-05-22 03:54:53,073 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data7/current/BP-1513808452-148.251.75.209-1684727668040] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:54:53,073 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data8/current/BP-1513808452-148.251.75.209-1684727668040] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:54:53,078 WARN [sync.3] wal.FSHLog(747): HDFS pipeline error detected. Found 1 replicas but expecting no less than 2 replicas. Requesting close of WAL. current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK]] 2023-05-22 03:54:53,079 WARN [sync.3] wal.FSHLog(718): Requesting log roll because of low replication, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK]] 2023-05-22 03:54:53,079 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(197): WAL FSHLog jenkins-hbase20.apache.org%2C37737%2C1684727669933:(num 1684727680876) roll requested 2023-05-22 03:54:53,082 WARN [Thread-640] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741840_1021 2023-05-22 03:54:53,085 WARN [Thread-640] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:44367,DS-75ff45d2-44e9-4a84-a4f0-954d0d3f85f0,DISK] 2023-05-22 03:54:53,091 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53028 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741841_1022]] datanode.DataXceiver(847): DataNode{data=FSDataset{dirpath='[/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data5/current, /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data6/current]'}, localName='127.0.0.1:45105', datanodeUuid='418679ee-d7f2-47eb-a5d7-1782bb1228b4', xmitsInProgress=0}:Exception transfering block BP-1513808452-148.251.75.209-1684727668040:blk_1073741841_1022 to mirror 127.0.0.1:37227: java.net.ConnectException: Connection refused 2023-05-22 03:54:53,091 WARN [Thread-640] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741841_1022 2023-05-22 03:54:53,092 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53028 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741841_1022]] datanode.DataXceiver(323): 127.0.0.1:45105:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:53028 dst: /127.0.0.1:45105 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:769) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:53,092 WARN [Thread-640] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK] 2023-05-22 03:54:53,100 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727680876 with entries=2, filesize=2.36 KB; new WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727693079 2023-05-22 03:54:53,100 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45105,DS-ecaf4275-1a8d-4716-8114-4b7b06871615,DISK], DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK]] 2023-05-22 03:54:53,100 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727680876 is not closed yet, will try archiving it next time 2023-05-22 03:54:55,544 WARN [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@10deff05] datanode.DataNode$DataTransfer(2503): DatanodeRegistration(127.0.0.1:44121, datanodeUuid=11861ffc-e31f-4fbf-b895-d9945de0e9e8, infoPort=39877, infoSecurePort=0, ipcPort=39439, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040):Failed to transfer BP-1513808452-148.251.75.209-1684727668040:blk_1073741839_1020 to 127.0.0.1:34581 got java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2431) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:57,090 WARN [Listener at localhost.localdomain/39439] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:54:57,092 WARN [ResponseProcessor for block BP-1513808452-148.251.75.209-1684727668040:blk_1073741842_1023] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-1513808452-148.251.75.209-1684727668040:blk_1073741842_1023 java.io.IOException: Bad response ERROR for BP-1513808452-148.251.75.209-1684727668040:blk_1073741842_1023 from datanode DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK] at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1120) 2023-05-22 03:54:57,092 WARN [DataStreamer for file /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727693079 block BP-1513808452-148.251.75.209-1684727668040:blk_1073741842_1023] hdfs.DataStreamer(1548): Error Recovery for BP-1513808452-148.251.75.209-1684727668040:blk_1073741842_1023 in pipeline [DatanodeInfoWithStorage[127.0.0.1:45105,DS-ecaf4275-1a8d-4716-8114-4b7b06871615,DISK], DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK]]: datanode 1(DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK]) is bad. 2023-05-22 03:54:57,092 WARN [PacketResponder: BP-1513808452-148.251.75.209-1684727668040:blk_1073741842_1023, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[127.0.0.1:44121]] datanode.BlockReceiver$PacketResponder(1486): IOException in BlockReceiver.run(): java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470) at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1630) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1565) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1478) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:57,093 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53034 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741842_1023]] datanode.DataXceiver(323): 127.0.0.1:45105:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:53034 dst: /127.0.0.1:45105 java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:57,095 INFO [Listener at localhost.localdomain/39439] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:54:57,199 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:42672 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741842_1023]] datanode.DataXceiver(323): 127.0.0.1:44121:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:42672 dst: /127.0.0.1:44121 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:57,202 WARN [BP-1513808452-148.251.75.209-1684727668040 heartbeating to localhost.localdomain/127.0.0.1:37617] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:54:57,202 WARN [BP-1513808452-148.251.75.209-1684727668040 heartbeating to localhost.localdomain/127.0.0.1:37617] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1513808452-148.251.75.209-1684727668040 (Datanode Uuid 11861ffc-e31f-4fbf-b895-d9945de0e9e8) service to localhost.localdomain/127.0.0.1:37617 2023-05-22 03:54:57,202 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data9/current/BP-1513808452-148.251.75.209-1684727668040] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:54:57,203 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data10/current/BP-1513808452-148.251.75.209-1684727668040] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:54:57,209 WARN [sync.1] wal.FSHLog(747): HDFS pipeline error detected. Found 1 replicas but expecting no less than 2 replicas. Requesting close of WAL. current pipeline: [DatanodeInfoWithStorage[127.0.0.1:45105,DS-ecaf4275-1a8d-4716-8114-4b7b06871615,DISK]] 2023-05-22 03:54:57,209 WARN [sync.1] wal.FSHLog(718): Requesting log roll because of low replication, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:45105,DS-ecaf4275-1a8d-4716-8114-4b7b06871615,DISK]] 2023-05-22 03:54:57,209 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(197): WAL FSHLog jenkins-hbase20.apache.org%2C37737%2C1684727669933:(num 1684727693079) roll requested 2023-05-22 03:54:57,216 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53042 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741843_1025]] datanode.DataXceiver(847): DataNode{data=FSDataset{dirpath='[/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data5/current, /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data6/current]'}, localName='127.0.0.1:45105', datanodeUuid='418679ee-d7f2-47eb-a5d7-1782bb1228b4', xmitsInProgress=0}:Exception transfering block BP-1513808452-148.251.75.209-1684727668040:blk_1073741843_1025 to mirror 127.0.0.1:44367: java.net.ConnectException: Connection refused 2023-05-22 03:54:57,216 WARN [Thread-652] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741843_1025 2023-05-22 03:54:57,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37737] regionserver.HRegion(9158): Flush requested on 65448373c7bf299ad9112f0ddcf80abc 2023-05-22 03:54:57,218 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53042 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741843_1025]] datanode.DataXceiver(323): 127.0.0.1:45105:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:53042 dst: /127.0.0.1:45105 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:769) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:57,219 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 65448373c7bf299ad9112f0ddcf80abc 1/1 column families, dataSize=7.36 KB heapSize=8.13 KB 2023-05-22 03:54:57,219 WARN [Thread-652] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:44367,DS-75ff45d2-44e9-4a84-a4f0-954d0d3f85f0,DISK] 2023-05-22 03:54:57,221 WARN [Thread-652] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741844_1026 2023-05-22 03:54:57,222 WARN [Thread-652] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK] 2023-05-22 03:54:57,227 WARN [Thread-652] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741845_1027 2023-05-22 03:54:57,229 WARN [Thread-652] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:34581,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK] 2023-05-22 03:54:57,230 WARN [Thread-654] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741846_1028 2023-05-22 03:54:57,231 WARN [Thread-652] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741847_1029 2023-05-22 03:54:57,231 WARN [Thread-654] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:44367,DS-75ff45d2-44e9-4a84-a4f0-954d0d3f85f0,DISK] 2023-05-22 03:54:57,231 WARN [Thread-652] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK] 2023-05-22 03:54:57,232 WARN [IPC Server handler 2 on default port 37617] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 2 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.hadoop.net.NetworkTopology 2023-05-22 03:54:57,233 WARN [IPC Server handler 2 on default port 37617] protocol.BlockStoragePolicy(161): Failed to place enough replicas: expected size is 1 but only 0 storage types can be selected (replication=2, selected=[], unavailable=[DISK], removed=[DISK], policy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}) 2023-05-22 03:54:57,233 WARN [Thread-654] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741848_1030 2023-05-22 03:54:57,233 WARN [IPC Server handler 2 on default port 37617] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 2 (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) All required storage types are unavailable: unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]} 2023-05-22 03:54:57,233 WARN [Thread-654] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:34581,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK] 2023-05-22 03:54:57,236 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53054 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741850_1032]] datanode.DataXceiver(847): DataNode{data=FSDataset{dirpath='[/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data5/current, /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data6/current]'}, localName='127.0.0.1:45105', datanodeUuid='418679ee-d7f2-47eb-a5d7-1782bb1228b4', xmitsInProgress=0}:Exception transfering block BP-1513808452-148.251.75.209-1684727668040:blk_1073741850_1032 to mirror 127.0.0.1:37227: java.net.ConnectException: Connection refused 2023-05-22 03:54:57,236 WARN [Thread-654] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741850_1032 2023-05-22 03:54:57,236 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53054 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741850_1032]] datanode.DataXceiver(323): 127.0.0.1:45105:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:53054 dst: /127.0.0.1:45105 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:769) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:57,237 WARN [Thread-654] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK] 2023-05-22 03:54:57,241 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727693079 with entries=12, filesize=12.95 KB; new WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727697209 2023-05-22 03:54:57,241 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45105,DS-ecaf4275-1a8d-4716-8114-4b7b06871615,DISK]] 2023-05-22 03:54:57,246 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727693079 is not closed yet, will try archiving it next time 2023-05-22 03:54:57,254 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53070 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741851_1033]] datanode.DataXceiver(847): DataNode{data=FSDataset{dirpath='[/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data5/current, /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data6/current]'}, localName='127.0.0.1:45105', datanodeUuid='418679ee-d7f2-47eb-a5d7-1782bb1228b4', xmitsInProgress=0}:Exception transfering block BP-1513808452-148.251.75.209-1684727668040:blk_1073741851_1033 to mirror 127.0.0.1:44121: java.net.ConnectException: Connection refused 2023-05-22 03:54:57,254 WARN [Thread-654] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741851_1033 2023-05-22 03:54:57,255 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53070 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741851_1033]] datanode.DataXceiver(323): 127.0.0.1:45105:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:53070 dst: /127.0.0.1:45105 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:769) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:57,255 WARN [Thread-654] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK] 2023-05-22 03:54:57,256 WARN [IPC Server handler 2 on default port 37617] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 2 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.hadoop.net.NetworkTopology 2023-05-22 03:54:57,256 WARN [IPC Server handler 2 on default port 37617] protocol.BlockStoragePolicy(161): Failed to place enough replicas: expected size is 1 but only 0 storage types can be selected (replication=2, selected=[], unavailable=[DISK], removed=[DISK], policy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}) 2023-05-22 03:54:57,256 WARN [IPC Server handler 2 on default port 37617] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 2 (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) All required storage types are unavailable: unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]} 2023-05-22 03:54:57,269 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=7.36 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/.tmp/info/a5d642f9b60045928879f06ce5156940 2023-05-22 03:54:57,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/.tmp/info/a5d642f9b60045928879f06ce5156940 as hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/info/a5d642f9b60045928879f06ce5156940 2023-05-22 03:54:57,293 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/info/a5d642f9b60045928879f06ce5156940, entries=5, sequenceid=12, filesize=10.0 K 2023-05-22 03:54:57,295 WARN [sync.4] wal.FSHLog(747): HDFS pipeline error detected. Found 1 replicas but expecting no less than 2 replicas. Requesting close of WAL. current pipeline: [DatanodeInfoWithStorage[127.0.0.1:45105,DS-ecaf4275-1a8d-4716-8114-4b7b06871615,DISK]] 2023-05-22 03:54:57,296 WARN [sync.4] wal.FSHLog(718): Requesting log roll because of low replication, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:45105,DS-ecaf4275-1a8d-4716-8114-4b7b06871615,DISK]] 2023-05-22 03:54:57,296 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~7.36 KB/7532, heapSize ~8.11 KB/8304, currentSize=7.35 KB/7531 for 65448373c7bf299ad9112f0ddcf80abc in 77ms, sequenceid=12, compaction requested=false 2023-05-22 03:54:57,296 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(197): WAL FSHLog jenkins-hbase20.apache.org%2C37737%2C1684727669933:(num 1684727697209) roll requested 2023-05-22 03:54:57,297 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 65448373c7bf299ad9112f0ddcf80abc: 2023-05-22 03:54:57,306 WARN [Thread-664] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741853_1035 2023-05-22 03:54:57,307 WARN [Thread-664] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:44367,DS-75ff45d2-44e9-4a84-a4f0-954d0d3f85f0,DISK] 2023-05-22 03:54:57,309 WARN [Thread-664] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741854_1036 2023-05-22 03:54:57,309 WARN [Thread-664] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK] 2023-05-22 03:54:57,313 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53098 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741855_1037]] datanode.DataXceiver(847): DataNode{data=FSDataset{dirpath='[/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data5/current, /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data6/current]'}, localName='127.0.0.1:45105', datanodeUuid='418679ee-d7f2-47eb-a5d7-1782bb1228b4', xmitsInProgress=0}:Exception transfering block BP-1513808452-148.251.75.209-1684727668040:blk_1073741855_1037 to mirror 127.0.0.1:34581: java.net.ConnectException: Connection refused 2023-05-22 03:54:57,313 WARN [Thread-664] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741855_1037 2023-05-22 03:54:57,313 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53098 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741855_1037]] datanode.DataXceiver(323): 127.0.0.1:45105:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:53098 dst: /127.0.0.1:45105 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:769) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:57,314 WARN [Thread-664] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:34581,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK] 2023-05-22 03:54:57,315 WARN [Thread-664] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741856_1038 2023-05-22 03:54:57,316 WARN [Thread-664] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK] 2023-05-22 03:54:57,324 WARN [IPC Server handler 1 on default port 37617] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 2 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.hadoop.net.NetworkTopology 2023-05-22 03:54:57,325 WARN [IPC Server handler 1 on default port 37617] protocol.BlockStoragePolicy(161): Failed to place enough replicas: expected size is 1 but only 0 storage types can be selected (replication=2, selected=[], unavailable=[DISK], removed=[DISK], policy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}) 2023-05-22 03:54:57,325 WARN [IPC Server handler 1 on default port 37617] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 2 (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) All required storage types are unavailable: unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]} 2023-05-22 03:54:57,340 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727697209 with entries=2, filesize=1.57 KB; new WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727697297 2023-05-22 03:54:57,340 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45105,DS-ecaf4275-1a8d-4716-8114-4b7b06871615,DISK]] 2023-05-22 03:54:57,340 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727697209 is not closed yet, will try archiving it next time 2023-05-22 03:54:57,340 INFO [WAL-Archive-0] wal.AbstractFSWAL(783): Archiving hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727680876 to hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/oldWALs/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727680876 2023-05-22 03:54:57,449 WARN [sync.1] wal.FSHLog(757): Too many consecutive RollWriter requests, it's a sign of the total number of live datanodes is lower than the tolerable replicas. 2023-05-22 03:54:57,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37737] regionserver.HRegion(9158): Flush requested on 65448373c7bf299ad9112f0ddcf80abc 2023-05-22 03:54:57,449 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 65448373c7bf299ad9112f0ddcf80abc 1/1 column families, dataSize=8.40 KB heapSize=9.25 KB 2023-05-22 03:54:57,469 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53106 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741858_1040]] datanode.DataXceiver(847): DataNode{data=FSDataset{dirpath='[/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data5/current, /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data6/current]'}, localName='127.0.0.1:45105', datanodeUuid='418679ee-d7f2-47eb-a5d7-1782bb1228b4', xmitsInProgress=0}:Exception transfering block BP-1513808452-148.251.75.209-1684727668040:blk_1073741858_1040 to mirror 127.0.0.1:34581: java.net.ConnectException: Connection refused 2023-05-22 03:54:57,469 WARN [Thread-669] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741858_1040 2023-05-22 03:54:57,469 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53106 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741858_1040]] datanode.DataXceiver(323): 127.0.0.1:45105:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:53106 dst: /127.0.0.1:45105 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:769) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:57,471 WARN [Thread-669] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:34581,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK] 2023-05-22 03:54:57,480 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53114 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741859_1041]] datanode.DataXceiver(847): DataNode{data=FSDataset{dirpath='[/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data5/current, /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data6/current]'}, localName='127.0.0.1:45105', datanodeUuid='418679ee-d7f2-47eb-a5d7-1782bb1228b4', xmitsInProgress=0}:Exception transfering block BP-1513808452-148.251.75.209-1684727668040:blk_1073741859_1041 to mirror 127.0.0.1:37227: java.net.ConnectException: Connection refused 2023-05-22 03:54:57,480 WARN [Thread-669] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741859_1041 2023-05-22 03:54:57,480 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53114 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741859_1041]] datanode.DataXceiver(323): 127.0.0.1:45105:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:53114 dst: /127.0.0.1:45105 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:769) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:57,481 WARN [Thread-669] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK] 2023-05-22 03:54:57,482 WARN [Thread-669] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741860_1042 2023-05-22 03:54:57,483 WARN [Thread-669] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:44367,DS-75ff45d2-44e9-4a84-a4f0-954d0d3f85f0,DISK] 2023-05-22 03:54:57,486 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53122 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741861_1043]] datanode.DataXceiver(847): DataNode{data=FSDataset{dirpath='[/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data5/current, /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data6/current]'}, localName='127.0.0.1:45105', datanodeUuid='418679ee-d7f2-47eb-a5d7-1782bb1228b4', xmitsInProgress=0}:Exception transfering block BP-1513808452-148.251.75.209-1684727668040:blk_1073741861_1043 to mirror 127.0.0.1:44121: java.net.ConnectException: Connection refused 2023-05-22 03:54:57,487 WARN [Thread-669] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741861_1043 2023-05-22 03:54:57,487 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:53122 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741861_1043]] datanode.DataXceiver(323): 127.0.0.1:45105:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:53122 dst: /127.0.0.1:45105 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:769) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:57,487 WARN [Thread-669] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK] 2023-05-22 03:54:57,488 WARN [IPC Server handler 4 on default port 37617] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 2 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.hadoop.net.NetworkTopology 2023-05-22 03:54:57,488 WARN [IPC Server handler 4 on default port 37617] protocol.BlockStoragePolicy(161): Failed to place enough replicas: expected size is 1 but only 0 storage types can be selected (replication=2, selected=[], unavailable=[DISK], removed=[DISK], policy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}) 2023-05-22 03:54:57,489 WARN [IPC Server handler 4 on default port 37617] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 2 (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) All required storage types are unavailable: unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]} 2023-05-22 03:54:57,655 WARN [Listener at localhost.localdomain/39439] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:54:57,657 WARN [Listener at localhost.localdomain/39439] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:54:57,659 INFO [Listener at localhost.localdomain/39439] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:54:57,667 INFO [Listener at localhost.localdomain/39439] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/java.io.tmpdir/Jetty_localhost_32771_datanode____dzifll/webapp 2023-05-22 03:54:57,773 INFO [Listener at localhost.localdomain/39439] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:32771 2023-05-22 03:54:57,790 WARN [Listener at localhost.localdomain/41983] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:54:57,896 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.40 KB at sequenceid=23 (bloomFilter=true), to=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/.tmp/info/4b4f0b357ca34c098a03964140332993 2023-05-22 03:54:57,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/.tmp/info/4b4f0b357ca34c098a03964140332993 as hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/info/4b4f0b357ca34c098a03964140332993 2023-05-22 03:54:57,919 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/info/4b4f0b357ca34c098a03964140332993, entries=7, sequenceid=23, filesize=12.1 K 2023-05-22 03:54:57,919 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2eab8f4e5df94174: Processing first storage report for DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46 from datanode a1e72017-c183-4eba-93c0-530451b9e3c0 2023-05-22 03:54:57,920 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~8.40 KB/8606, heapSize ~9.23 KB/9456, currentSize=0 B/0 for 65448373c7bf299ad9112f0ddcf80abc in 471ms, sequenceid=23, compaction requested=false 2023-05-22 03:54:57,921 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 65448373c7bf299ad9112f0ddcf80abc: 2023-05-22 03:54:57,921 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2eab8f4e5df94174: from storage DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46 node DatanodeRegistration(127.0.0.1:39133, datanodeUuid=a1e72017-c183-4eba-93c0-530451b9e3c0, infoPort=40237, infoSecurePort=0, ipcPort=41983, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040), blocks: 7, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-05-22 03:54:57,921 DEBUG [MemStoreFlusher.0] regionserver.ConstantSizeRegionSplitPolicy(109): Should split because info size=22.1 K, sizeToCheck=16.0 K 2023-05-22 03:54:57,921 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2eab8f4e5df94174: Processing first storage report for DS-3e0e64b5-429a-4785-9e0c-2d361afbcc63 from datanode a1e72017-c183-4eba-93c0-530451b9e3c0 2023-05-22 03:54:57,921 DEBUG [MemStoreFlusher.0] regionserver.IncreasingToUpperBoundRegionSplitPolicy(84): regionsWithCommonTable=1 2023-05-22 03:54:57,921 DEBUG [MemStoreFlusher.0] regionserver.StoreUtils(129): cannot split hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/info/4b4f0b357ca34c098a03964140332993 because midkey is the same as first or last row 2023-05-22 03:54:57,921 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2eab8f4e5df94174: from storage DS-3e0e64b5-429a-4785-9e0c-2d361afbcc63 node DatanodeRegistration(127.0.0.1:39133, datanodeUuid=a1e72017-c183-4eba-93c0-530451b9e3c0, infoPort=40237, infoSecurePort=0, ipcPort=41983, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040), blocks: 7, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-05-22 03:54:58,264 WARN [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@2a05e7a8] datanode.DataNode$DataTransfer(2503): DatanodeRegistration(127.0.0.1:45105, datanodeUuid=418679ee-d7f2-47eb-a5d7-1782bb1228b4, infoPort=33161, infoSecurePort=0, ipcPort=35455, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040):Failed to transfer BP-1513808452-148.251.75.209-1684727668040:blk_1073741842_1024 to 127.0.0.1:44121 got java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2431) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:58,264 WARN [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@7d61ad65] datanode.DataNode$DataTransfer(2503): DatanodeRegistration(127.0.0.1:45105, datanodeUuid=418679ee-d7f2-47eb-a5d7-1782bb1228b4, infoPort=33161, infoSecurePort=0, ipcPort=35455, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040):Failed to transfer BP-1513808452-148.251.75.209-1684727668040:blk_1073741852_1034 to 127.0.0.1:44121 got java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2431) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:54:58,853 WARN [master/jenkins-hbase20:0:becomeActiveMaster.append-pool-0] wal.FSHLog$RingBufferEventHandler(1203): Append sequenceId=91, requesting roll of WAL java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:54:58,853 DEBUG [master:store-WAL-Roller] wal.AbstractWALRoller(197): WAL FSHLog jenkins-hbase20.apache.org%2C46365%2C1684727668643:(num 1684727668777) roll requested 2023-05-22 03:54:58,858 WARN [Thread-711] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741863_1045 2023-05-22 03:54:58,858 ERROR [ProcExecTimeout] helpers.MarkerIgnoringBase(151): Failed to delete pids=[4, 7, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=91, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1205) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1078) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:979) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.lang.Thread.run(Thread.java:750) Caused by: java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:54:58,859 ERROR [ProcExecTimeout] procedure2.TimeoutExecutorThread(124): Ignoring pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner exception: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=91, requesting roll of WAL java.io.UncheckedIOException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=91, requesting roll of WAL at org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore.delete(RegionProcedureStore.java:423) at org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner.periodicExecute(CompletedProcedureCleaner.java:135) at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.executeInMemoryChore(TimeoutExecutorThread.java:122) at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.execDelayedProcedure(TimeoutExecutorThread.java:101) at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:68) Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=91, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1205) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1078) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:979) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.lang.Thread.run(Thread.java:750) Caused by: java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:54:58,859 WARN [Thread-711] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:44367,DS-75ff45d2-44e9-4a84-a4f0-954d0d3f85f0,DISK] 2023-05-22 03:54:58,864 WARN [master:store-WAL-Roller] wal.FSHLog(374): Failed sync-before-close but no outstanding appends; closing WALorg.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=91, requesting roll of WAL 2023-05-22 03:54:58,864 INFO [master:store-WAL-Roller] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/WALs/jenkins-hbase20.apache.org,46365,1684727668643/jenkins-hbase20.apache.org%2C46365%2C1684727668643.1684727668777 with entries=88, filesize=43.75 KB; new WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/WALs/jenkins-hbase20.apache.org,46365,1684727668643/jenkins-hbase20.apache.org%2C46365%2C1684727668643.1684727698853 2023-05-22 03:54:58,864 DEBUG [master:store-WAL-Roller] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45105,DS-ecaf4275-1a8d-4716-8114-4b7b06871615,DISK], DatanodeInfoWithStorage[127.0.0.1:39133,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK]] 2023-05-22 03:54:58,864 DEBUG [master:store-WAL-Roller] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/WALs/jenkins-hbase20.apache.org,46365,1684727668643/jenkins-hbase20.apache.org%2C46365%2C1684727668643.1684727668777 is not closed yet, will try archiving it next time 2023-05-22 03:54:58,864 WARN [Close-WAL-Writer-0] wal.AbstractProtobufLogWriter(237): Failed to write trailer, non-fatal, continuing... java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:54:58,865 WARN [Close-WAL-Writer-0] wal.FSHLog(466): Riding over failed WAL close of hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/WALs/jenkins-hbase20.apache.org,46365,1684727668643/jenkins-hbase20.apache.org%2C46365%2C1684727668643.1684727668777; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:54:59,266 WARN [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@76b5b3dc] datanode.DataNode$DataTransfer(2503): DatanodeRegistration(127.0.0.1:45105, datanodeUuid=418679ee-d7f2-47eb-a5d7-1782bb1228b4, infoPort=33161, infoSecurePort=0, ipcPort=35455, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040):Failed to transfer BP-1513808452-148.251.75.209-1684727668040:blk_1073741849_1031 to 127.0.0.1:44367 got java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2431) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:05,921 WARN [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@2a947177] datanode.DataNode$DataTransfer(2503): DatanodeRegistration(127.0.0.1:39133, datanodeUuid=a1e72017-c183-4eba-93c0-530451b9e3c0, infoPort=40237, infoSecurePort=0, ipcPort=41983, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040):Failed to transfer BP-1513808452-148.251.75.209-1684727668040:blk_1073741830_1006 to 127.0.0.1:44121 got java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2431) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:05,922 WARN [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@b162fda] datanode.DataNode$DataTransfer(2503): DatanodeRegistration(127.0.0.1:39133, datanodeUuid=a1e72017-c183-4eba-93c0-530451b9e3c0, infoPort=40237, infoSecurePort=0, ipcPort=41983, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040):Failed to transfer BP-1513808452-148.251.75.209-1684727668040:blk_1073741828_1004 to 127.0.0.1:44367 got java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2431) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:07,918 WARN [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@5c72d600] datanode.DataNode$DataTransfer(2503): DatanodeRegistration(127.0.0.1:39133, datanodeUuid=a1e72017-c183-4eba-93c0-530451b9e3c0, infoPort=40237, infoSecurePort=0, ipcPort=41983, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040):Failed to transfer BP-1513808452-148.251.75.209-1684727668040:blk_1073741827_1003 to 127.0.0.1:44121 got java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2431) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:07,918 WARN [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@fc61bc3] datanode.DataNode$DataTransfer(2503): DatanodeRegistration(127.0.0.1:39133, datanodeUuid=a1e72017-c183-4eba-93c0-530451b9e3c0, infoPort=40237, infoSecurePort=0, ipcPort=41983, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040):Failed to transfer BP-1513808452-148.251.75.209-1684727668040:blk_1073741825_1001 to 127.0.0.1:44367 got java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2431) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:10,918 WARN [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@6d0cc85] datanode.DataNode$DataTransfer(2503): DatanodeRegistration(127.0.0.1:39133, datanodeUuid=a1e72017-c183-4eba-93c0-530451b9e3c0, infoPort=40237, infoSecurePort=0, ipcPort=41983, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040):Failed to transfer BP-1513808452-148.251.75.209-1684727668040:blk_1073741826_1002 to 127.0.0.1:44367 got java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2431) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:11,920 WARN [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@768a582c] datanode.DataNode$DataTransfer(2503): DatanodeRegistration(127.0.0.1:39133, datanodeUuid=a1e72017-c183-4eba-93c0-530451b9e3c0, infoPort=40237, infoSecurePort=0, ipcPort=41983, storageInfo=lv=-57;cid=testClusterID;nsid=1702157036;c=1684727668040):Failed to transfer BP-1513808452-148.251.75.209-1684727668040:blk_1073741835_1011 to 127.0.0.1:44367 got java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2431) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:16,643 WARN [Thread-735] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741865_1047 2023-05-22 03:55:16,643 WARN [Thread-735] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK] 2023-05-22 03:55:16,660 INFO [Listener at localhost.localdomain/41983] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727697297 with entries=3, filesize=1.89 KB; new WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727716636 2023-05-22 03:55:16,660 DEBUG [Listener at localhost.localdomain/41983] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45105,DS-ecaf4275-1a8d-4716-8114-4b7b06871615,DISK], DatanodeInfoWithStorage[127.0.0.1:39133,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK]] 2023-05-22 03:55:16,661 DEBUG [Listener at localhost.localdomain/41983] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727697297 is not closed yet, will try archiving it next time 2023-05-22 03:55:16,661 INFO [WAL-Archive-0] wal.AbstractFSWAL(783): Archiving hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727693079 to hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/oldWALs/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727693079 2023-05-22 03:55:16,672 INFO [WAL-Archive-0] wal.AbstractFSWAL(783): Archiving hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727697209 to hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/oldWALs/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727697209 2023-05-22 03:55:16,673 INFO [sync.1] wal.FSHLog(774): LowReplication-Roller was enabled. 2023-05-22 03:55:16,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37737] regionserver.HRegion(9158): Flush requested on 65448373c7bf299ad9112f0ddcf80abc 2023-05-22 03:55:16,682 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 65448373c7bf299ad9112f0ddcf80abc 1/1 column families, dataSize=7.36 KB heapSize=8.13 KB 2023-05-22 03:55:16,687 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-05-22 03:55:16,687 INFO [Listener at localhost.localdomain/41983] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-05-22 03:55:16,687 DEBUG [Listener at localhost.localdomain/41983] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x268de280 to 127.0.0.1:58195 2023-05-22 03:55:16,687 DEBUG [Listener at localhost.localdomain/41983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:55:16,687 DEBUG [Listener at localhost.localdomain/41983] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-05-22 03:55:16,687 DEBUG [Listener at localhost.localdomain/41983] util.JVMClusterUtil(257): Found active master hash=410092229, stopped=false 2023-05-22 03:55:16,687 INFO [Listener at localhost.localdomain/41983] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,46365,1684727668643 2023-05-22 03:55:16,699 WARN [Thread-742] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741867_1049 2023-05-22 03:55:16,699 WARN [Thread-742] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK] 2023-05-22 03:55:16,702 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): regionserver:37737-0x101714e37b60005, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-05-22 03:55:16,703 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): regionserver:45231-0x101714e37b60001, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-05-22 03:55:16,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45231-0x101714e37b60001, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:55:16,707 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-05-22 03:55:16,707 INFO [Listener at localhost.localdomain/41983] procedure2.ProcedureExecutor(629): Stopping 2023-05-22 03:55:16,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37737-0x101714e37b60005, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:55:16,707 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:55:16,707 DEBUG [Listener at localhost.localdomain/41983] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x64aeb685 to 127.0.0.1:58195 2023-05-22 03:55:16,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:55:16,708 DEBUG [Listener at localhost.localdomain/41983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:55:16,708 INFO [Listener at localhost.localdomain/41983] regionserver.HRegionServer(2295): ***** STOPPING region server 'jenkins-hbase20.apache.org,45231,1684727668683' ***** 2023-05-22 03:55:16,708 INFO [Listener at localhost.localdomain/41983] regionserver.HRegionServer(2309): STOPPED: Shutdown requested 2023-05-22 03:55:16,708 INFO [Listener at localhost.localdomain/41983] regionserver.HRegionServer(2295): ***** STOPPING region server 'jenkins-hbase20.apache.org,37737,1684727669933' ***** 2023-05-22 03:55:16,708 INFO [Listener at localhost.localdomain/41983] regionserver.HRegionServer(2309): STOPPED: Shutdown requested 2023-05-22 03:55:16,708 INFO [RS:0;jenkins-hbase20:45231] regionserver.HeapMemoryManager(220): Stopping 2023-05-22 03:55:16,708 INFO [RS:0;jenkins-hbase20:45231] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-05-22 03:55:16,708 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-05-22 03:55:16,708 INFO [RS:0;jenkins-hbase20:45231] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-05-22 03:55:16,709 INFO [RS:1;jenkins-hbase20:37737] regionserver.HeapMemoryManager(220): Stopping 2023-05-22 03:55:16,709 INFO [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(3303): Received CLOSE for d0d8f2fdf86d7a316e2ac72baf57c403 2023-05-22 03:55:16,709 INFO [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,45231,1684727668683 2023-05-22 03:55:16,709 DEBUG [RS:0;jenkins-hbase20:45231] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c3de25d to 127.0.0.1:58195 2023-05-22 03:55:16,709 DEBUG [RS:0;jenkins-hbase20:45231] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:55:16,710 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing d0d8f2fdf86d7a316e2ac72baf57c403, disabling compactions & flushes 2023-05-22 03:55:16,710 INFO [RS:0;jenkins-hbase20:45231] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-05-22 03:55:16,710 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. 2023-05-22 03:55:16,710 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. 2023-05-22 03:55:16,710 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. after waiting 0 ms 2023-05-22 03:55:16,710 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. 2023-05-22 03:55:16,710 INFO [RS:0;jenkins-hbase20:45231] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-05-22 03:55:16,710 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing d0d8f2fdf86d7a316e2ac72baf57c403 1/1 column families, dataSize=78 B heapSize=488 B 2023-05-22 03:55:16,710 INFO [RS:0;jenkins-hbase20:45231] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-05-22 03:55:16,711 INFO [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(3303): Received CLOSE for 1588230740 2023-05-22 03:55:16,711 INFO [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-05-22 03:55:16,711 DEBUG [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, d0d8f2fdf86d7a316e2ac72baf57c403=hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403.} 2023-05-22 03:55:16,711 WARN [RS:0;jenkins-hbase20:45231.append-pool-0] wal.FSHLog$RingBufferEventHandler(1203): Append sequenceId=7, requesting roll of WAL java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:16,712 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(197): WAL FSHLog jenkins-hbase20.apache.org%2C45231%2C1684727668683:(num 1684727669082) roll requested 2023-05-22 03:55:16,711 DEBUG [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(1504): Waiting on 1588230740, d0d8f2fdf86d7a316e2ac72baf57c403 2023-05-22 03:55:16,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for d0d8f2fdf86d7a316e2ac72baf57c403: 2023-05-22 03:55:16,712 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=7.36 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/.tmp/info/e0d9ddaa2a0a44cabf31cf1ac2893957 2023-05-22 03:55:16,712 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-05-22 03:55:16,713 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-05-22 03:55:16,713 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-05-22 03:55:16,713 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-05-22 03:55:16,713 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-05-22 03:55:16,713 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] helpers.MarkerIgnoringBase(159): ***** ABORTING region server jenkins-hbase20.apache.org,45231,1684727668683: Unrecoverable exception while closing hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. ***** org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=7, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1205) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1078) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:979) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.lang.Thread.run(Thread.java:750) Caused by: java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:16,713 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=2.93 KB heapSize=5.45 KB 2023-05-22 03:55:16,713 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-05-22 03:55:16,714 WARN [RS_OPEN_META-regionserver/jenkins-hbase20:0-0.append-pool-0] wal.FSHLog$RingBufferEventHandler(1203): Append sequenceId=15, requesting roll of WAL java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:16,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-05-22 03:55:16,718 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-05-22 03:55:16,718 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2539): Abort already in progress. Ignoring the current request with reason: Unrecoverable exception while closing hbase:meta,,1.1588230740 2023-05-22 03:55:16,720 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-05-22 03:55:16,720 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-05-22 03:55:16,720 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-05-22 03:55:16,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2555): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1030750208, "init": 524288000, "max": 2051014656, "used": 313319432 }, "NonHeapMemoryUsage": { "committed": 133718016, "init": 2555904, "max": -1, "used": 130998976 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [], "beans": [], "beans": [] } 2023-05-22 03:55:16,727 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1810824824_17 at /127.0.0.1:56492 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741869_1051]] datanode.DataXceiver(847): DataNode{data=FSDataset{dirpath='[/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data3/current, /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data4/current]'}, localName='127.0.0.1:39133', datanodeUuid='a1e72017-c183-4eba-93c0-530451b9e3c0', xmitsInProgress=0}:Exception transfering block BP-1513808452-148.251.75.209-1684727668040:blk_1073741869_1051 to mirror 127.0.0.1:44121: java.net.ConnectException: Connection refused 2023-05-22 03:55:16,727 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1810824824_17 at /127.0.0.1:56492 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741869_1051]] datanode.DataXceiver(323): 127.0.0.1:39133:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:56492 dst: /127.0.0.1:39133 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:769) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:16,727 WARN [Thread-750] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741869_1051 2023-05-22 03:55:16,728 WARN [Thread-750] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK] 2023-05-22 03:55:16,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46365] master.MasterRpcServices(609): jenkins-hbase20.apache.org,45231,1684727668683 reported a fatal error: ***** ABORTING region server jenkins-hbase20.apache.org,45231,1684727668683: Unrecoverable exception while closing hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. ***** Cause: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=7, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1205) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1078) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:979) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.lang.Thread.run(Thread.java:750) Caused by: java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:16,729 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/.tmp/info/e0d9ddaa2a0a44cabf31cf1ac2893957 as hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/info/e0d9ddaa2a0a44cabf31cf1ac2893957 2023-05-22 03:55:16,742 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/info/e0d9ddaa2a0a44cabf31cf1ac2893957, entries=7, sequenceid=33, filesize=12.1 K 2023-05-22 03:55:16,743 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~7.36 KB/7532, heapSize ~8.11 KB/8304, currentSize=3.15 KB/3228 for 65448373c7bf299ad9112f0ddcf80abc in 61ms, sequenceid=33, compaction requested=true 2023-05-22 03:55:16,744 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 65448373c7bf299ad9112f0ddcf80abc: 2023-05-22 03:55:16,744 WARN [regionserver/jenkins-hbase20:0.logRoller] wal.FSHLog(374): Failed sync-before-close but no outstanding appends; closing WALorg.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=7, requesting roll of WAL 2023-05-22 03:55:16,744 DEBUG [MemStoreFlusher.0] regionserver.ConstantSizeRegionSplitPolicy(109): Should split because info size=34.2 K, sizeToCheck=16.0 K 2023-05-22 03:55:16,744 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683/jenkins-hbase20.apache.org%2C45231%2C1684727668683.1684727669082 with entries=3, filesize=601 B; new WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683/jenkins-hbase20.apache.org%2C45231%2C1684727668683.1684727716712 2023-05-22 03:55:16,744 DEBUG [MemStoreFlusher.0] regionserver.IncreasingToUpperBoundRegionSplitPolicy(84): regionsWithCommonTable=1 2023-05-22 03:55:16,744 DEBUG [MemStoreFlusher.0] regionserver.StoreUtils(129): cannot split hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/info/e0d9ddaa2a0a44cabf31cf1ac2893957 because midkey is the same as first or last row 2023-05-22 03:55:16,745 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-05-22 03:55:16,746 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45105,DS-ecaf4275-1a8d-4716-8114-4b7b06871615,DISK], DatanodeInfoWithStorage[127.0.0.1:39133,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK]] 2023-05-22 03:55:16,746 INFO [RS:1;jenkins-hbase20:37737] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-05-22 03:55:16,746 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683/jenkins-hbase20.apache.org%2C45231%2C1684727668683.1684727669082 is not closed yet, will try archiving it next time 2023-05-22 03:55:16,746 WARN [Close-WAL-Writer-0] wal.AbstractProtobufLogWriter(237): Failed to write trailer, non-fatal, continuing... java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:16,746 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(197): WAL FSHLog jenkins-hbase20.apache.org%2C45231%2C1684727668683.meta:.meta(num 1684727669257) roll requested 2023-05-22 03:55:16,746 WARN [Close-WAL-Writer-0] wal.FSHLog(466): Riding over failed WAL close of hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683/jenkins-hbase20.apache.org%2C45231%2C1684727668683.1684727669082; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:16,746 INFO [RS:1;jenkins-hbase20:37737] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-05-22 03:55:16,746 INFO [RS:1;jenkins-hbase20:37737] regionserver.HRegionServer(3303): Received CLOSE for 65448373c7bf299ad9112f0ddcf80abc 2023-05-22 03:55:16,750 INFO [RS:1;jenkins-hbase20:37737] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,37737,1684727669933 2023-05-22 03:55:16,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 65448373c7bf299ad9112f0ddcf80abc, disabling compactions & flushes 2023-05-22 03:55:16,751 DEBUG [RS:1;jenkins-hbase20:37737] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4adee753 to 127.0.0.1:58195 2023-05-22 03:55:16,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc. 2023-05-22 03:55:16,751 DEBUG [RS:1;jenkins-hbase20:37737] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:55:16,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc. 2023-05-22 03:55:16,751 INFO [RS:1;jenkins-hbase20:37737] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-05-22 03:55:16,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc. after waiting 0 ms 2023-05-22 03:55:16,751 DEBUG [RS:1;jenkins-hbase20:37737] regionserver.HRegionServer(1478): Online Regions={65448373c7bf299ad9112f0ddcf80abc=TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc.} 2023-05-22 03:55:16,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc. 2023-05-22 03:55:16,751 DEBUG [RS:1;jenkins-hbase20:37737] regionserver.HRegionServer(1504): Waiting on 65448373c7bf299ad9112f0ddcf80abc 2023-05-22 03:55:16,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 65448373c7bf299ad9112f0ddcf80abc 1/1 column families, dataSize=3.15 KB heapSize=3.63 KB 2023-05-22 03:55:16,755 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1810824824_17 at /127.0.0.1:56520 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741871_1053]] datanode.DataXceiver(847): DataNode{data=FSDataset{dirpath='[/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data3/current, /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data4/current]'}, localName='127.0.0.1:39133', datanodeUuid='a1e72017-c183-4eba-93c0-530451b9e3c0', xmitsInProgress=0}:Exception transfering block BP-1513808452-148.251.75.209-1684727668040:blk_1073741871_1053 to mirror 127.0.0.1:44121: java.net.ConnectException: Connection refused 2023-05-22 03:55:16,755 WARN [Thread-759] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741871_1053 2023-05-22 03:55:16,761 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1810824824_17 at /127.0.0.1:56520 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741871_1053]] datanode.DataXceiver(323): 127.0.0.1:39133:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:56520 dst: /127.0.0.1:39133 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:769) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:16,763 WARN [Thread-759] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK] 2023-05-22 03:55:16,772 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:42976 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741873_1055]] datanode.DataXceiver(847): DataNode{data=FSDataset{dirpath='[/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data5/current, /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data6/current]'}, localName='127.0.0.1:45105', datanodeUuid='418679ee-d7f2-47eb-a5d7-1782bb1228b4', xmitsInProgress=0}:Exception transfering block BP-1513808452-148.251.75.209-1684727668040:blk_1073741873_1055 to mirror 127.0.0.1:44121: java.net.ConnectException: Connection refused 2023-05-22 03:55:16,772 WARN [Thread-761] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741873_1055 2023-05-22 03:55:16,772 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-190359387_17 at /127.0.0.1:42976 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741873_1055]] datanode.DataXceiver(323): 127.0.0.1:45105:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:42976 dst: /127.0.0.1:45105 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:769) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:16,774 WARN [Thread-761] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK] 2023-05-22 03:55:16,786 WARN [regionserver/jenkins-hbase20:0.logRoller] wal.FSHLog(374): Failed sync-before-close but no outstanding appends; closing WALorg.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=15, requesting roll of WAL 2023-05-22 03:55:16,786 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683/jenkins-hbase20.apache.org%2C45231%2C1684727668683.meta.1684727669257.meta with entries=11, filesize=3.69 KB; new WAL /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683/jenkins-hbase20.apache.org%2C45231%2C1684727668683.meta.1684727716746.meta 2023-05-22 03:55:16,786 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45105,DS-ecaf4275-1a8d-4716-8114-4b7b06871615,DISK], DatanodeInfoWithStorage[127.0.0.1:39133,DS-c8817a9e-74c2-40d7-81e0-27d3ffe75c46,DISK]] 2023-05-22 03:55:16,786 WARN [Close-WAL-Writer-0] wal.AbstractProtobufLogWriter(237): Failed to write trailer, non-fatal, continuing... java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:16,786 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683/jenkins-hbase20.apache.org%2C45231%2C1684727668683.meta.1684727669257.meta is not closed yet, will try archiving it next time 2023-05-22 03:55:16,786 WARN [Close-WAL-Writer-0] wal.FSHLog(466): Riding over failed WAL close of hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683/jenkins-hbase20.apache.org%2C45231%2C1684727668683.meta.1684727669257.meta; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:37227,DS-c2b559d0-b47c-4ad6-a580-d9681724a793,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:16,794 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.15 KB at sequenceid=39 (bloomFilter=true), to=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/.tmp/info/f3798080ef204a8c90aa452256fd3b9d 2023-05-22 03:55:16,802 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/.tmp/info/f3798080ef204a8c90aa452256fd3b9d as hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/info/f3798080ef204a8c90aa452256fd3b9d 2023-05-22 03:55:16,809 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/info/f3798080ef204a8c90aa452256fd3b9d, entries=3, sequenceid=39, filesize=7.9 K 2023-05-22 03:55:16,810 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.15 KB/3228, heapSize ~3.61 KB/3696, currentSize=0 B/0 for 65448373c7bf299ad9112f0ddcf80abc in 59ms, sequenceid=39, compaction requested=true 2023-05-22 03:55:16,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/data/default/TestLogRolling-testLogRollOnDatanodeDeath/65448373c7bf299ad9112f0ddcf80abc/recovered.edits/42.seqid, newMaxSeqId=42, maxSeqId=1 2023-05-22 03:55:16,818 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc. 2023-05-22 03:55:16,818 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 65448373c7bf299ad9112f0ddcf80abc: 2023-05-22 03:55:16,819 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed TestLogRolling-testLogRollOnDatanodeDeath,,1684727670027.65448373c7bf299ad9112f0ddcf80abc. 2023-05-22 03:55:16,912 INFO [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(3303): Received CLOSE for 1588230740 2023-05-22 03:55:16,913 INFO [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(3303): Received CLOSE for d0d8f2fdf86d7a316e2ac72baf57c403 2023-05-22 03:55:16,913 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-05-22 03:55:16,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing d0d8f2fdf86d7a316e2ac72baf57c403, disabling compactions & flushes 2023-05-22 03:55:16,913 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-05-22 03:55:16,913 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-05-22 03:55:16,913 DEBUG [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(1504): Waiting on 1588230740, d0d8f2fdf86d7a316e2ac72baf57c403 2023-05-22 03:55:16,913 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-05-22 03:55:16,913 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. 2023-05-22 03:55:16,913 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-05-22 03:55:16,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. 2023-05-22 03:55:16,913 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-05-22 03:55:16,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. after waiting 0 ms 2023-05-22 03:55:16,913 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2539): Abort already in progress. Ignoring the current request with reason: Unrecoverable exception while closing hbase:meta,,1.1588230740 2023-05-22 03:55:16,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. 2023-05-22 03:55:16,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for d0d8f2fdf86d7a316e2ac72baf57c403: 2023-05-22 03:55:16,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2539): Abort already in progress. Ignoring the current request with reason: Unrecoverable exception while closing hbase:namespace,,1684727669324.d0d8f2fdf86d7a316e2ac72baf57c403. 2023-05-22 03:55:16,951 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-05-22 03:55:16,952 INFO [RS:1;jenkins-hbase20:37737] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,37737,1684727669933; all regions closed. 2023-05-22 03:55:16,952 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933 2023-05-22 03:55:16,997 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-05-22 03:55:16,997 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-05-22 03:55:17,021 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-05-22 03:55:17,021 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-05-22 03:55:17,073 INFO [WAL-Archive-0] wal.AbstractFSWAL(783): Archiving hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,37737,1684727669933/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727697297 to hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/oldWALs/jenkins-hbase20.apache.org%2C37737%2C1684727669933.1684727697297 2023-05-22 03:55:17,076 DEBUG [RS:1;jenkins-hbase20:37737] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/oldWALs 2023-05-22 03:55:17,076 INFO [RS:1;jenkins-hbase20:37737] wal.AbstractFSWAL(1031): Closed WAL: FSHLog jenkins-hbase20.apache.org%2C37737%2C1684727669933:(num 1684727716636) 2023-05-22 03:55:17,076 DEBUG [RS:1;jenkins-hbase20:37737] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:55:17,077 INFO [RS:1;jenkins-hbase20:37737] regionserver.LeaseManager(133): Closed leases 2023-05-22 03:55:17,077 INFO [RS:1;jenkins-hbase20:37737] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-05-22 03:55:17,077 INFO [RS:1;jenkins-hbase20:37737] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-05-22 03:55:17,077 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-05-22 03:55:17,077 INFO [RS:1;jenkins-hbase20:37737] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-05-22 03:55:17,077 INFO [RS:1;jenkins-hbase20:37737] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-05-22 03:55:17,078 INFO [RS:1;jenkins-hbase20:37737] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:37737 2023-05-22 03:55:17,087 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:55:17,087 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): regionserver:45231-0x101714e37b60001, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,37737,1684727669933 2023-05-22 03:55:17,087 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): regionserver:37737-0x101714e37b60005, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,37737,1684727669933 2023-05-22 03:55:17,087 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): regionserver:37737-0x101714e37b60005, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:55:17,087 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): regionserver:45231-0x101714e37b60001, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:55:17,088 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,37737,1684727669933] 2023-05-22 03:55:17,088 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,37737,1684727669933; numProcessing=1 2023-05-22 03:55:17,089 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,37737,1684727669933 already deleted, retry=false 2023-05-22 03:55:17,089 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,37737,1684727669933 expired; onlineServers=1 2023-05-22 03:55:17,113 INFO [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(1499): We were exiting though online regions are not empty, because some regions failed closing 2023-05-22 03:55:17,113 INFO [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,45231,1684727668683; all regions closed. 2023-05-22 03:55:17,115 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683 2023-05-22 03:55:17,124 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/WALs/jenkins-hbase20.apache.org,45231,1684727668683 2023-05-22 03:55:17,138 DEBUG [RS:0;jenkins-hbase20:45231] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:55:17,138 INFO [RS:0;jenkins-hbase20:45231] regionserver.LeaseManager(133): Closed leases 2023-05-22 03:55:17,138 INFO [RS:0;jenkins-hbase20:45231] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-05-22 03:55:17,138 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-05-22 03:55:17,139 INFO [RS:0;jenkins-hbase20:45231] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:45231 2023-05-22 03:55:17,143 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:55:17,143 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): regionserver:45231-0x101714e37b60001, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,45231,1684727668683 2023-05-22 03:55:17,145 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,45231,1684727668683] 2023-05-22 03:55:17,145 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,45231,1684727668683; numProcessing=2 2023-05-22 03:55:17,145 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,45231,1684727668683 already deleted, retry=false 2023-05-22 03:55:17,146 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,45231,1684727668683 expired; onlineServers=0 2023-05-22 03:55:17,146 INFO [RegionServerTracker-0] regionserver.HRegionServer(2295): ***** STOPPING region server 'jenkins-hbase20.apache.org,46365,1684727668643' ***** 2023-05-22 03:55:17,146 INFO [RegionServerTracker-0] regionserver.HRegionServer(2309): STOPPED: Cluster shutdown set; onlineServer=0 2023-05-22 03:55:17,146 DEBUG [M:0;jenkins-hbase20:46365] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c4e1513, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-05-22 03:55:17,147 INFO [M:0;jenkins-hbase20:46365] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,46365,1684727668643 2023-05-22 03:55:17,147 INFO [M:0;jenkins-hbase20:46365] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,46365,1684727668643; all regions closed. 2023-05-22 03:55:17,147 DEBUG [M:0;jenkins-hbase20:46365] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:55:17,147 DEBUG [M:0;jenkins-hbase20:46365] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-05-22 03:55:17,147 DEBUG [M:0;jenkins-hbase20:46365] cleaner.HFileCleaner(317): Stopping file delete threads 2023-05-22 03:55:17,147 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727668855] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727668855,5,FailOnTimeoutGroup] 2023-05-22 03:55:17,147 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-05-22 03:55:17,147 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727668858] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727668858,5,FailOnTimeoutGroup] 2023-05-22 03:55:17,147 INFO [M:0;jenkins-hbase20:46365] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-05-22 03:55:17,147 INFO [M:0;jenkins-hbase20:46365] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-05-22 03:55:17,148 INFO [M:0;jenkins-hbase20:46365] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-05-22 03:55:17,148 DEBUG [M:0;jenkins-hbase20:46365] master.HMaster(1512): Stopping service threads 2023-05-22 03:55:17,148 INFO [M:0;jenkins-hbase20:46365] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-05-22 03:55:17,149 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-05-22 03:55:17,149 ERROR [M:0;jenkins-hbase20:46365] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-05-22 03:55:17,149 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:55:17,149 INFO [M:0;jenkins-hbase20:46365] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-05-22 03:55:17,149 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-05-22 03:55:17,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:55:17,150 DEBUG [M:0;jenkins-hbase20:46365] zookeeper.ZKUtil(398): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-05-22 03:55:17,150 WARN [M:0;jenkins-hbase20:46365] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-05-22 03:55:17,150 INFO [M:0;jenkins-hbase20:46365] assignment.AssignmentManager(315): Stopping assignment manager 2023-05-22 03:55:17,150 INFO [M:0;jenkins-hbase20:46365] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-05-22 03:55:17,150 DEBUG [M:0;jenkins-hbase20:46365] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-05-22 03:55:17,150 INFO [M:0;jenkins-hbase20:46365] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:55:17,151 DEBUG [M:0;jenkins-hbase20:46365] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:55:17,151 DEBUG [M:0;jenkins-hbase20:46365] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-05-22 03:55:17,151 DEBUG [M:0;jenkins-hbase20:46365] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:55:17,151 INFO [M:0;jenkins-hbase20:46365] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=38.12 KB heapSize=45.77 KB 2023-05-22 03:55:17,167 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1237576890_17 at /127.0.0.1:56544 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741875_1057]] datanode.DataXceiver(847): DataNode{data=FSDataset{dirpath='[/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data3/current, /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data4/current]'}, localName='127.0.0.1:39133', datanodeUuid='a1e72017-c183-4eba-93c0-530451b9e3c0', xmitsInProgress=0}:Exception transfering block BP-1513808452-148.251.75.209-1684727668040:blk_1073741875_1057 to mirror 127.0.0.1:44121: java.net.ConnectException: Connection refused 2023-05-22 03:55:17,167 WARN [Thread-775] hdfs.DataStreamer(1658): Abandoning BP-1513808452-148.251.75.209-1684727668040:blk_1073741875_1057 2023-05-22 03:55:17,167 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1237576890_17 at /127.0.0.1:56544 [Receiving block BP-1513808452-148.251.75.209-1684727668040:blk_1073741875_1057]] datanode.DataXceiver(323): 127.0.0.1:39133:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:56544 dst: /127.0.0.1:39133 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:769) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:17,168 WARN [Thread-775] hdfs.DataStreamer(1663): Excluding datanode DatanodeInfoWithStorage[127.0.0.1:44121,DS-0b79ddfc-3d5c-47e4-a449-55a3b3c99cbc,DISK] 2023-05-22 03:55:17,189 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): regionserver:37737-0x101714e37b60005, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:55:17,189 INFO [RS:1;jenkins-hbase20:37737] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,37737,1684727669933; zookeeper connection closed. 2023-05-22 03:55:17,189 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): regionserver:37737-0x101714e37b60005, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:55:17,189 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@a3e8d9a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@a3e8d9a 2023-05-22 03:55:17,190 INFO [M:0;jenkins-hbase20:46365] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.12 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/a297b042f0664f678d17ff4557bc1380 2023-05-22 03:55:17,199 DEBUG [M:0;jenkins-hbase20:46365] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/a297b042f0664f678d17ff4557bc1380 as hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/a297b042f0664f678d17ff4557bc1380 2023-05-22 03:55:17,205 INFO [M:0;jenkins-hbase20:46365] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37617/user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/a297b042f0664f678d17ff4557bc1380, entries=11, sequenceid=92, filesize=7.0 K 2023-05-22 03:55:17,206 INFO [M:0;jenkins-hbase20:46365] regionserver.HRegion(2948): Finished flush of dataSize ~38.12 KB/39035, heapSize ~45.75 KB/46848, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 55ms, sequenceid=92, compaction requested=false 2023-05-22 03:55:17,209 INFO [M:0;jenkins-hbase20:46365] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:55:17,209 DEBUG [M:0;jenkins-hbase20:46365] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:55:17,215 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/d25a44f5-146f-1a37-d499-b177bd8e7b91/MasterData/WALs/jenkins-hbase20.apache.org,46365,1684727668643 2023-05-22 03:55:17,219 INFO [M:0;jenkins-hbase20:46365] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-05-22 03:55:17,219 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-05-22 03:55:17,220 INFO [M:0;jenkins-hbase20:46365] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:46365 2023-05-22 03:55:17,222 DEBUG [M:0;jenkins-hbase20:46365] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,46365,1684727668643 already deleted, retry=false 2023-05-22 03:55:17,289 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): regionserver:45231-0x101714e37b60001, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:55:17,289 INFO [RS:0;jenkins-hbase20:45231] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,45231,1684727668683; zookeeper connection closed. 2023-05-22 03:55:17,289 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): regionserver:45231-0x101714e37b60001, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:55:17,290 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2a6e1844] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2a6e1844 2023-05-22 03:55:17,291 INFO [Listener at localhost.localdomain/41983] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 2 regionserver(s) complete 2023-05-22 03:55:17,390 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:55:17,390 DEBUG [Listener at localhost.localdomain/44663-EventThread] zookeeper.ZKWatcher(600): master:46365-0x101714e37b60000, quorum=127.0.0.1:58195, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:55:17,390 INFO [M:0;jenkins-hbase20:46365] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,46365,1684727668643; zookeeper connection closed. 2023-05-22 03:55:17,391 WARN [Listener at localhost.localdomain/41983] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:55:17,397 INFO [Listener at localhost.localdomain/41983] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:55:17,500 WARN [BP-1513808452-148.251.75.209-1684727668040 heartbeating to localhost.localdomain/127.0.0.1:37617] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:55:17,500 WARN [BP-1513808452-148.251.75.209-1684727668040 heartbeating to localhost.localdomain/127.0.0.1:37617] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1513808452-148.251.75.209-1684727668040 (Datanode Uuid a1e72017-c183-4eba-93c0-530451b9e3c0) service to localhost.localdomain/127.0.0.1:37617 2023-05-22 03:55:17,501 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data3/current/BP-1513808452-148.251.75.209-1684727668040] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:55:17,501 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data4/current/BP-1513808452-148.251.75.209-1684727668040] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:55:17,502 WARN [Listener at localhost.localdomain/41983] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:55:17,507 INFO [Listener at localhost.localdomain/41983] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:55:17,610 WARN [BP-1513808452-148.251.75.209-1684727668040 heartbeating to localhost.localdomain/127.0.0.1:37617] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:55:17,610 WARN [BP-1513808452-148.251.75.209-1684727668040 heartbeating to localhost.localdomain/127.0.0.1:37617] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1513808452-148.251.75.209-1684727668040 (Datanode Uuid 418679ee-d7f2-47eb-a5d7-1782bb1228b4) service to localhost.localdomain/127.0.0.1:37617 2023-05-22 03:55:17,611 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data5/current/BP-1513808452-148.251.75.209-1684727668040] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:55:17,611 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/cluster_0f3a75de-a148-a7de-6736-5a6b7c88e52c/dfs/data/data6/current/BP-1513808452-148.251.75.209-1684727668040] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:55:17,621 INFO [Listener at localhost.localdomain/41983] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-05-22 03:55:17,737 INFO [Listener at localhost.localdomain/41983] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-05-22 03:55:17,784 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-05-22 03:55:17,796 INFO [Listener at localhost.localdomain/41983] hbase.ResourceChecker(175): after: regionserver.wal.TestLogRolling#testLogRollOnDatanodeDeath Thread=78 (was 52) Potentially hanging thread: nioEventLoopGroup-10-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-16-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.1@localhost.localdomain:37617 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-3-worker-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: Listener at localhost.localdomain/41983 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-17-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (551965115) connection to localhost.localdomain/127.0.0.1:37617 from jenkins.hfs.2 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: Abort regionserver monitor java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: nioEventLoopGroup-17-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-16-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-16-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (551965115) connection to localhost.localdomain/127.0.0.1:37617 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost.localdomain:37617 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'DataNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: IPC Client (551965115) connection to localhost.localdomain/127.0.0.1:37617 from jenkins.hfs.1 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-17-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost.localdomain:37617 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (551965115) connection to localhost.localdomain/127.0.0.1:37617 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-10-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-3-worker-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=472 (was 441) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=154 (was 135) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 170), AvailableMemoryMB=7386 (was 7326) - AvailableMemoryMB LEAK? - 2023-05-22 03:55:17,811 INFO [Listener at localhost.localdomain/41983] hbase.ResourceChecker(147): before: regionserver.wal.TestLogRolling#testLogRollOnPipelineRestart Thread=78, OpenFileDescriptor=472, MaxFileDescriptor=60000, SystemLoadAverage=154, ProcessCount=168, AvailableMemoryMB=7386 2023-05-22 03:55:17,811 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-05-22 03:55:17,812 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/hadoop.log.dir so I do NOT create it in target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc 2023-05-22 03:55:17,812 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a035619b-c4ed-ccc8-538d-b20092165f5e/hadoop.tmp.dir so I do NOT create it in target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc 2023-05-22 03:55:17,812 INFO [Listener at localhost.localdomain/41983] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35, deleteOnExit=true 2023-05-22 03:55:17,812 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-05-22 03:55:17,812 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/test.cache.data in system properties and HBase conf 2023-05-22 03:55:17,812 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/hadoop.tmp.dir in system properties and HBase conf 2023-05-22 03:55:17,813 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/hadoop.log.dir in system properties and HBase conf 2023-05-22 03:55:17,813 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/mapreduce.cluster.local.dir in system properties and HBase conf 2023-05-22 03:55:17,813 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-05-22 03:55:17,813 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-05-22 03:55:17,813 DEBUG [Listener at localhost.localdomain/41983] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-05-22 03:55:17,814 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-05-22 03:55:17,814 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-05-22 03:55:17,814 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-05-22 03:55:17,814 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-05-22 03:55:17,814 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-05-22 03:55:17,814 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-05-22 03:55:17,815 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-05-22 03:55:17,815 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/dfs.journalnode.edits.dir in system properties and HBase conf 2023-05-22 03:55:17,815 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-05-22 03:55:17,815 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/nfs.dump.dir in system properties and HBase conf 2023-05-22 03:55:17,815 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/java.io.tmpdir in system properties and HBase conf 2023-05-22 03:55:17,815 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/dfs.journalnode.edits.dir in system properties and HBase conf 2023-05-22 03:55:17,815 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-05-22 03:55:17,816 INFO [Listener at localhost.localdomain/41983] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-05-22 03:55:17,817 WARN [Listener at localhost.localdomain/41983] blockmanagement.DatanodeManager(362): The given interval for marking stale datanode = 30000, which is larger than heartbeat expire interval 20000. 2023-05-22 03:55:17,819 WARN [Listener at localhost.localdomain/41983] conf.Configuration(1701): No unit for dfs.heartbeat.interval(1) assuming SECONDS 2023-05-22 03:55:17,819 WARN [Listener at localhost.localdomain/41983] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-05-22 03:55:17,856 WARN [Listener at localhost.localdomain/41983] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:55:17,859 INFO [Listener at localhost.localdomain/41983] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:55:17,865 INFO [Listener at localhost.localdomain/41983] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/java.io.tmpdir/Jetty_localhost_localdomain_45379_hdfs____v1vd3i/webapp 2023-05-22 03:55:17,940 INFO [Listener at localhost.localdomain/41983] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:45379 2023-05-22 03:55:17,941 WARN [Listener at localhost.localdomain/41983] blockmanagement.DatanodeManager(362): The given interval for marking stale datanode = 30000, which is larger than heartbeat expire interval 20000. 2023-05-22 03:55:17,942 WARN [Listener at localhost.localdomain/41983] conf.Configuration(1701): No unit for dfs.heartbeat.interval(1) assuming SECONDS 2023-05-22 03:55:17,943 WARN [Listener at localhost.localdomain/41983] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-05-22 03:55:17,970 WARN [Listener at localhost.localdomain/37871] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:55:17,980 WARN [Listener at localhost.localdomain/37871] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:55:17,982 WARN [Listener at localhost.localdomain/37871] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:55:17,983 INFO [Listener at localhost.localdomain/37871] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:55:17,989 INFO [Listener at localhost.localdomain/37871] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/java.io.tmpdir/Jetty_localhost_38483_datanode____ciy26p/webapp 2023-05-22 03:55:17,999 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-05-22 03:55:18,069 INFO [Listener at localhost.localdomain/37871] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38483 2023-05-22 03:55:18,075 WARN [Listener at localhost.localdomain/45799] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:55:18,094 WARN [Listener at localhost.localdomain/45799] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:55:18,097 WARN [Listener at localhost.localdomain/45799] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:55:18,098 INFO [Listener at localhost.localdomain/45799] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:55:18,105 INFO [Listener at localhost.localdomain/45799] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/java.io.tmpdir/Jetty_localhost_44541_datanode____.5aszl3/webapp 2023-05-22 03:55:18,152 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xde631354d89b306f: Processing first storage report for DS-085d43d7-a480-48a8-8078-32697106b7a8 from datanode 6eb1cf74-c837-4445-a555-7e1f0435d520 2023-05-22 03:55:18,152 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xde631354d89b306f: from storage DS-085d43d7-a480-48a8-8078-32697106b7a8 node DatanodeRegistration(127.0.0.1:35699, datanodeUuid=6eb1cf74-c837-4445-a555-7e1f0435d520, infoPort=45101, infoSecurePort=0, ipcPort=45799, storageInfo=lv=-57;cid=testClusterID;nsid=1219854411;c=1684727717821), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:55:18,152 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xde631354d89b306f: Processing first storage report for DS-60858ccf-ea05-49f2-aeec-a20984865bec from datanode 6eb1cf74-c837-4445-a555-7e1f0435d520 2023-05-22 03:55:18,152 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xde631354d89b306f: from storage DS-60858ccf-ea05-49f2-aeec-a20984865bec node DatanodeRegistration(127.0.0.1:35699, datanodeUuid=6eb1cf74-c837-4445-a555-7e1f0435d520, infoPort=45101, infoSecurePort=0, ipcPort=45799, storageInfo=lv=-57;cid=testClusterID;nsid=1219854411;c=1684727717821), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:55:18,190 INFO [Listener at localhost.localdomain/45799] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44541 2023-05-22 03:55:18,198 WARN [Listener at localhost.localdomain/35103] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:55:18,260 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc80618c6116e449e: Processing first storage report for DS-86ac7614-eb39-435c-9344-7c2530a2a13f from datanode 5181a363-6d97-47ef-86f3-4730977e7fb9 2023-05-22 03:55:18,260 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc80618c6116e449e: from storage DS-86ac7614-eb39-435c-9344-7c2530a2a13f node DatanodeRegistration(127.0.0.1:36747, datanodeUuid=5181a363-6d97-47ef-86f3-4730977e7fb9, infoPort=40261, infoSecurePort=0, ipcPort=35103, storageInfo=lv=-57;cid=testClusterID;nsid=1219854411;c=1684727717821), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:55:18,260 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc80618c6116e449e: Processing first storage report for DS-b9e2f754-2dd9-4cee-aad4-2ae7b2c446a1 from datanode 5181a363-6d97-47ef-86f3-4730977e7fb9 2023-05-22 03:55:18,260 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc80618c6116e449e: from storage DS-b9e2f754-2dd9-4cee-aad4-2ae7b2c446a1 node DatanodeRegistration(127.0.0.1:36747, datanodeUuid=5181a363-6d97-47ef-86f3-4730977e7fb9, infoPort=40261, infoSecurePort=0, ipcPort=35103, storageInfo=lv=-57;cid=testClusterID;nsid=1219854411;c=1684727717821), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:55:18,307 DEBUG [Listener at localhost.localdomain/35103] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc 2023-05-22 03:55:18,309 INFO [Listener at localhost.localdomain/35103] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/zookeeper_0, clientPort=49349, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-05-22 03:55:18,310 INFO [Listener at localhost.localdomain/35103] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=49349 2023-05-22 03:55:18,310 INFO [Listener at localhost.localdomain/35103] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:55:18,311 INFO [Listener at localhost.localdomain/35103] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:55:18,326 INFO [Listener at localhost.localdomain/35103] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d with version=8 2023-05-22 03:55:18,326 INFO [Listener at localhost.localdomain/35103] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/hbase-staging 2023-05-22 03:55:18,328 INFO [Listener at localhost.localdomain/35103] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-05-22 03:55:18,328 INFO [Listener at localhost.localdomain/35103] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:55:18,328 INFO [Listener at localhost.localdomain/35103] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-05-22 03:55:18,328 INFO [Listener at localhost.localdomain/35103] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-05-22 03:55:18,328 INFO [Listener at localhost.localdomain/35103] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:55:18,328 INFO [Listener at localhost.localdomain/35103] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-05-22 03:55:18,328 INFO [Listener at localhost.localdomain/35103] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-05-22 03:55:18,329 INFO [Listener at localhost.localdomain/35103] ipc.NettyRpcServer(120): Bind to /148.251.75.209:44087 2023-05-22 03:55:18,330 INFO [Listener at localhost.localdomain/35103] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:55:18,331 INFO [Listener at localhost.localdomain/35103] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:55:18,332 INFO [Listener at localhost.localdomain/35103] zookeeper.RecoverableZooKeeper(93): Process identifier=master:44087 connecting to ZooKeeper ensemble=127.0.0.1:49349 2023-05-22 03:55:18,337 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:440870x0, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-05-22 03:55:18,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:44087-0x101714ef9d00000 connected 2023-05-22 03:55:18,352 DEBUG [Listener at localhost.localdomain/35103] zookeeper.ZKUtil(164): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:55:18,352 DEBUG [Listener at localhost.localdomain/35103] zookeeper.ZKUtil(164): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:55:18,353 DEBUG [Listener at localhost.localdomain/35103] zookeeper.ZKUtil(164): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-05-22 03:55:18,353 DEBUG [Listener at localhost.localdomain/35103] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44087 2023-05-22 03:55:18,354 DEBUG [Listener at localhost.localdomain/35103] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44087 2023-05-22 03:55:18,354 DEBUG [Listener at localhost.localdomain/35103] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44087 2023-05-22 03:55:18,355 DEBUG [Listener at localhost.localdomain/35103] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44087 2023-05-22 03:55:18,355 DEBUG [Listener at localhost.localdomain/35103] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44087 2023-05-22 03:55:18,355 INFO [Listener at localhost.localdomain/35103] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d, hbase.cluster.distributed=false 2023-05-22 03:55:18,365 INFO [Listener at localhost.localdomain/35103] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-05-22 03:55:18,365 INFO [Listener at localhost.localdomain/35103] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:55:18,365 INFO [Listener at localhost.localdomain/35103] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-05-22 03:55:18,365 INFO [Listener at localhost.localdomain/35103] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-05-22 03:55:18,366 INFO [Listener at localhost.localdomain/35103] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:55:18,366 INFO [Listener at localhost.localdomain/35103] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-05-22 03:55:18,366 INFO [Listener at localhost.localdomain/35103] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-05-22 03:55:18,367 INFO [Listener at localhost.localdomain/35103] ipc.NettyRpcServer(120): Bind to /148.251.75.209:40121 2023-05-22 03:55:18,367 INFO [Listener at localhost.localdomain/35103] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-05-22 03:55:18,368 DEBUG [Listener at localhost.localdomain/35103] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-05-22 03:55:18,369 INFO [Listener at localhost.localdomain/35103] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:55:18,369 INFO [Listener at localhost.localdomain/35103] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:55:18,370 INFO [Listener at localhost.localdomain/35103] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40121 connecting to ZooKeeper ensemble=127.0.0.1:49349 2023-05-22 03:55:18,373 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): regionserver:401210x0, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-05-22 03:55:18,374 DEBUG [Listener at localhost.localdomain/35103] zookeeper.ZKUtil(164): regionserver:401210x0, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:55:18,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:40121-0x101714ef9d00001 connected 2023-05-22 03:55:18,375 DEBUG [Listener at localhost.localdomain/35103] zookeeper.ZKUtil(164): regionserver:40121-0x101714ef9d00001, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:55:18,375 DEBUG [Listener at localhost.localdomain/35103] zookeeper.ZKUtil(164): regionserver:40121-0x101714ef9d00001, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-05-22 03:55:18,379 DEBUG [Listener at localhost.localdomain/35103] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40121 2023-05-22 03:55:18,379 DEBUG [Listener at localhost.localdomain/35103] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40121 2023-05-22 03:55:18,380 DEBUG [Listener at localhost.localdomain/35103] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40121 2023-05-22 03:55:18,380 DEBUG [Listener at localhost.localdomain/35103] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40121 2023-05-22 03:55:18,382 DEBUG [Listener at localhost.localdomain/35103] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40121 2023-05-22 03:55:18,384 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,44087,1684727718327 2023-05-22 03:55:18,399 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-05-22 03:55:18,399 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,44087,1684727718327 2023-05-22 03:55:18,400 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-05-22 03:55:18,400 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): regionserver:40121-0x101714ef9d00001, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-05-22 03:55:18,401 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:55:18,401 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-05-22 03:55:18,402 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,44087,1684727718327 from backup master directory 2023-05-22 03:55:18,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-05-22 03:55:18,403 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,44087,1684727718327 2023-05-22 03:55:18,403 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-05-22 03:55:18,403 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-05-22 03:55:18,404 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,44087,1684727718327 2023-05-22 03:55:18,421 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/hbase.id with ID: b6c03522-07cc-4156-a5b6-504a78bbfbee 2023-05-22 03:55:18,433 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:55:18,435 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:55:18,453 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x6eec54ca to 127.0.0.1:49349 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:55:18,458 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54c3ff38, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:55:18,458 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-05-22 03:55:18,459 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-05-22 03:55:18,459 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:55:18,461 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/data/master/store-tmp 2023-05-22 03:55:18,477 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:55:18,478 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-05-22 03:55:18,478 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:55:18,478 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:55:18,478 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-05-22 03:55:18,478 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:55:18,478 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:55:18,478 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:55:18,478 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/WALs/jenkins-hbase20.apache.org,44087,1684727718327 2023-05-22 03:55:18,485 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C44087%2C1684727718327, suffix=, logDir=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/WALs/jenkins-hbase20.apache.org,44087,1684727718327, archiveDir=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/oldWALs, maxLogs=10 2023-05-22 03:55:18,498 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/WALs/jenkins-hbase20.apache.org,44087,1684727718327/jenkins-hbase20.apache.org%2C44087%2C1684727718327.1684727718485 2023-05-22 03:55:18,498 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36747,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK], DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] 2023-05-22 03:55:18,498 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:55:18,498 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:55:18,499 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:55:18,499 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:55:18,501 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:55:18,502 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-05-22 03:55:18,503 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-05-22 03:55:18,503 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:55:18,504 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:55:18,505 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:55:18,507 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:55:18,509 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:55:18,510 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=716484, jitterRate=-0.08894394338130951}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:55:18,510 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:55:18,510 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-05-22 03:55:18,511 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-05-22 03:55:18,511 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-05-22 03:55:18,511 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-05-22 03:55:18,512 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-05-22 03:55:18,512 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-05-22 03:55:18,512 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-05-22 03:55:18,513 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-05-22 03:55:18,514 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-05-22 03:55:18,524 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-05-22 03:55:18,524 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-22 03:55:18,525 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-05-22 03:55:18,526 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-05-22 03:55:18,526 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-05-22 03:55:18,528 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:55:18,528 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-05-22 03:55:18,528 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-05-22 03:55:18,529 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-05-22 03:55:18,530 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-05-22 03:55:18,530 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): regionserver:40121-0x101714ef9d00001, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-05-22 03:55:18,530 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:55:18,531 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,44087,1684727718327, sessionid=0x101714ef9d00000, setting cluster-up flag (Was=false) 2023-05-22 03:55:18,535 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:55:18,538 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-05-22 03:55:18,539 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,44087,1684727718327 2023-05-22 03:55:18,542 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:55:18,545 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-05-22 03:55:18,545 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,44087,1684727718327 2023-05-22 03:55:18,546 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/.hbase-snapshot/.tmp 2023-05-22 03:55:18,552 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-05-22 03:55:18,553 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:55:18,553 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:55:18,553 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:55:18,553 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:55:18,553 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-05-22 03:55:18,553 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:55:18,553 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-05-22 03:55:18,553 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:55:18,559 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1684727748558 2023-05-22 03:55:18,559 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-05-22 03:55:18,559 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-05-22 03:55:18,560 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-05-22 03:55:18,560 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-05-22 03:55:18,560 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-05-22 03:55:18,560 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-05-22 03:55:18,560 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-05-22 03:55:18,561 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-05-22 03:55:18,561 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-05-22 03:55:18,561 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-05-22 03:55:18,561 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-05-22 03:55:18,561 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-05-22 03:55:18,561 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-05-22 03:55:18,561 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-05-22 03:55:18,562 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727718562,5,FailOnTimeoutGroup] 2023-05-22 03:55:18,562 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727718562,5,FailOnTimeoutGroup] 2023-05-22 03:55:18,562 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-05-22 03:55:18,562 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-05-22 03:55:18,562 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-05-22 03:55:18,562 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-05-22 03:55:18,563 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-05-22 03:55:18,572 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-05-22 03:55:18,573 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-05-22 03:55:18,573 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d 2023-05-22 03:55:18,584 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:55:18,584 INFO [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(951): ClusterId : b6c03522-07cc-4156-a5b6-504a78bbfbee 2023-05-22 03:55:18,585 DEBUG [RS:0;jenkins-hbase20:40121] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-05-22 03:55:18,586 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-05-22 03:55:18,587 DEBUG [RS:0;jenkins-hbase20:40121] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-05-22 03:55:18,587 DEBUG [RS:0;jenkins-hbase20:40121] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-05-22 03:55:18,587 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/meta/1588230740/info 2023-05-22 03:55:18,588 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-05-22 03:55:18,588 DEBUG [RS:0;jenkins-hbase20:40121] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-05-22 03:55:18,589 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:55:18,589 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-05-22 03:55:18,589 DEBUG [RS:0;jenkins-hbase20:40121] zookeeper.ReadOnlyZKClient(139): Connect 0x3bdc7e65 to 127.0.0.1:49349 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:55:18,591 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:55:18,592 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-05-22 03:55:18,593 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:55:18,593 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-05-22 03:55:18,594 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/meta/1588230740/table 2023-05-22 03:55:18,595 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-05-22 03:55:18,595 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:55:18,596 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/meta/1588230740 2023-05-22 03:55:18,596 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/meta/1588230740 2023-05-22 03:55:18,599 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-05-22 03:55:18,600 DEBUG [RS:0;jenkins-hbase20:40121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3fe3fdc6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:55:18,600 DEBUG [RS:0;jenkins-hbase20:40121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ae1745e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-05-22 03:55:18,600 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-05-22 03:55:18,602 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:55:18,603 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=823176, jitterRate=0.04672341048717499}}}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-05-22 03:55:18,603 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-05-22 03:55:18,603 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-05-22 03:55:18,603 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-05-22 03:55:18,603 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-05-22 03:55:18,603 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-05-22 03:55:18,603 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-05-22 03:55:18,604 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-05-22 03:55:18,604 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-05-22 03:55:18,605 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-05-22 03:55:18,605 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-05-22 03:55:18,605 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-05-22 03:55:18,607 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-05-22 03:55:18,608 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-05-22 03:55:18,612 DEBUG [RS:0;jenkins-hbase20:40121] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:40121 2023-05-22 03:55:18,612 INFO [RS:0;jenkins-hbase20:40121] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-05-22 03:55:18,612 INFO [RS:0;jenkins-hbase20:40121] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-05-22 03:55:18,612 DEBUG [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(1022): About to register with Master. 2023-05-22 03:55:18,612 INFO [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(2809): reportForDuty to master=jenkins-hbase20.apache.org,44087,1684727718327 with isa=jenkins-hbase20.apache.org/148.251.75.209:40121, startcode=1684727718365 2023-05-22 03:55:18,613 DEBUG [RS:0;jenkins-hbase20:40121] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-05-22 03:55:18,616 INFO [RS-EventLoopGroup-8-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60245, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=RegionServerStatusService 2023-05-22 03:55:18,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44087] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,40121,1684727718365 2023-05-22 03:55:18,618 DEBUG [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d 2023-05-22 03:55:18,618 DEBUG [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:37871 2023-05-22 03:55:18,618 DEBUG [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-05-22 03:55:18,619 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:55:18,620 DEBUG [RS:0;jenkins-hbase20:40121] zookeeper.ZKUtil(162): regionserver:40121-0x101714ef9d00001, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,40121,1684727718365 2023-05-22 03:55:18,620 WARN [RS:0;jenkins-hbase20:40121] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-05-22 03:55:18,620 INFO [RS:0;jenkins-hbase20:40121] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:55:18,620 DEBUG [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(1946): logDir=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365 2023-05-22 03:55:18,620 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,40121,1684727718365] 2023-05-22 03:55:18,624 DEBUG [RS:0;jenkins-hbase20:40121] zookeeper.ZKUtil(162): regionserver:40121-0x101714ef9d00001, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,40121,1684727718365 2023-05-22 03:55:18,624 DEBUG [RS:0;jenkins-hbase20:40121] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-05-22 03:55:18,624 INFO [RS:0;jenkins-hbase20:40121] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-05-22 03:55:18,626 INFO [RS:0;jenkins-hbase20:40121] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-05-22 03:55:18,626 INFO [RS:0;jenkins-hbase20:40121] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-05-22 03:55:18,626 INFO [RS:0;jenkins-hbase20:40121] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:55:18,626 INFO [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-05-22 03:55:18,628 INFO [RS:0;jenkins-hbase20:40121] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-05-22 03:55:18,628 DEBUG [RS:0;jenkins-hbase20:40121] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:55:18,628 DEBUG [RS:0;jenkins-hbase20:40121] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:55:18,628 DEBUG [RS:0;jenkins-hbase20:40121] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:55:18,628 DEBUG [RS:0;jenkins-hbase20:40121] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:55:18,628 DEBUG [RS:0;jenkins-hbase20:40121] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:55:18,628 DEBUG [RS:0;jenkins-hbase20:40121] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-05-22 03:55:18,628 DEBUG [RS:0;jenkins-hbase20:40121] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:55:18,628 DEBUG [RS:0;jenkins-hbase20:40121] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:55:18,628 DEBUG [RS:0;jenkins-hbase20:40121] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:55:18,628 DEBUG [RS:0;jenkins-hbase20:40121] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:55:18,629 INFO [RS:0;jenkins-hbase20:40121] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-05-22 03:55:18,629 INFO [RS:0;jenkins-hbase20:40121] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-05-22 03:55:18,630 INFO [RS:0;jenkins-hbase20:40121] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-05-22 03:55:18,642 INFO [RS:0;jenkins-hbase20:40121] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-05-22 03:55:18,642 INFO [RS:0;jenkins-hbase20:40121] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40121,1684727718365-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:55:18,650 INFO [RS:0;jenkins-hbase20:40121] regionserver.Replication(203): jenkins-hbase20.apache.org,40121,1684727718365 started 2023-05-22 03:55:18,650 INFO [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,40121,1684727718365, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:40121, sessionid=0x101714ef9d00001 2023-05-22 03:55:18,650 DEBUG [RS:0;jenkins-hbase20:40121] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-05-22 03:55:18,650 DEBUG [RS:0;jenkins-hbase20:40121] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,40121,1684727718365 2023-05-22 03:55:18,650 DEBUG [RS:0;jenkins-hbase20:40121] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,40121,1684727718365' 2023-05-22 03:55:18,650 DEBUG [RS:0;jenkins-hbase20:40121] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:55:18,651 DEBUG [RS:0;jenkins-hbase20:40121] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:55:18,651 DEBUG [RS:0;jenkins-hbase20:40121] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-05-22 03:55:18,651 DEBUG [RS:0;jenkins-hbase20:40121] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-05-22 03:55:18,651 DEBUG [RS:0;jenkins-hbase20:40121] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,40121,1684727718365 2023-05-22 03:55:18,651 DEBUG [RS:0;jenkins-hbase20:40121] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,40121,1684727718365' 2023-05-22 03:55:18,651 DEBUG [RS:0;jenkins-hbase20:40121] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-05-22 03:55:18,652 DEBUG [RS:0;jenkins-hbase20:40121] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-05-22 03:55:18,652 DEBUG [RS:0;jenkins-hbase20:40121] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-05-22 03:55:18,652 INFO [RS:0;jenkins-hbase20:40121] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-05-22 03:55:18,652 INFO [RS:0;jenkins-hbase20:40121] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-05-22 03:55:18,756 INFO [RS:0;jenkins-hbase20:40121] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C40121%2C1684727718365, suffix=, logDir=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365, archiveDir=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/oldWALs, maxLogs=32 2023-05-22 03:55:18,759 DEBUG [jenkins-hbase20:44087] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-05-22 03:55:18,761 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,40121,1684727718365, state=OPENING 2023-05-22 03:55:18,765 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-05-22 03:55:18,766 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:55:18,766 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,40121,1684727718365}] 2023-05-22 03:55:18,766 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-05-22 03:55:18,772 INFO [RS:0;jenkins-hbase20:40121] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727718758 2023-05-22 03:55:18,772 DEBUG [RS:0;jenkins-hbase20:40121] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK], DatanodeInfoWithStorage[127.0.0.1:36747,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK]] 2023-05-22 03:55:18,923 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,40121,1684727718365 2023-05-22 03:55:18,923 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-05-22 03:55:18,928 INFO [RS-EventLoopGroup-9-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54656, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-05-22 03:55:18,933 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-05-22 03:55:18,933 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:55:18,936 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C40121%2C1684727718365.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365, archiveDir=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/oldWALs, maxLogs=32 2023-05-22 03:55:18,953 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.meta.1684727718938.meta 2023-05-22 03:55:18,953 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36747,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK], DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] 2023-05-22 03:55:18,953 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:55:18,953 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-05-22 03:55:18,953 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-05-22 03:55:18,954 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-05-22 03:55:18,954 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-05-22 03:55:18,954 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:55:18,954 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-05-22 03:55:18,954 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-05-22 03:55:18,957 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-05-22 03:55:18,958 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/meta/1588230740/info 2023-05-22 03:55:18,958 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/meta/1588230740/info 2023-05-22 03:55:18,958 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-05-22 03:55:18,959 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:55:18,959 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-05-22 03:55:18,960 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:55:18,960 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:55:18,960 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-05-22 03:55:18,961 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:55:18,961 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-05-22 03:55:18,961 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/meta/1588230740/table 2023-05-22 03:55:18,961 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/meta/1588230740/table 2023-05-22 03:55:18,962 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-05-22 03:55:18,962 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:55:18,963 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/meta/1588230740 2023-05-22 03:55:18,964 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/meta/1588230740 2023-05-22 03:55:18,966 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-05-22 03:55:18,967 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-05-22 03:55:18,968 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=788222, jitterRate=0.0022765398025512695}}}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-05-22 03:55:18,968 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-05-22 03:55:18,970 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1684727718923 2023-05-22 03:55:18,974 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for hbase:meta,,1.1588230740 2023-05-22 03:55:18,975 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-05-22 03:55:18,976 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,40121,1684727718365, state=OPEN 2023-05-22 03:55:18,977 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-05-22 03:55:18,977 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-05-22 03:55:18,980 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-05-22 03:55:18,980 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,40121,1684727718365 in 211 msec 2023-05-22 03:55:18,983 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-05-22 03:55:18,983 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 375 msec 2023-05-22 03:55:18,985 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 433 msec 2023-05-22 03:55:18,985 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1684727718985, completionTime=-1 2023-05-22 03:55:18,985 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-05-22 03:55:18,985 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-05-22 03:55:18,988 DEBUG [hconnection-0x2cb410f1-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-05-22 03:55:18,990 INFO [RS-EventLoopGroup-9-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54664, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-05-22 03:55:18,992 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-05-22 03:55:18,992 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1684727778992 2023-05-22 03:55:18,992 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1684727838992 2023-05-22 03:55:18,992 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-05-22 03:55:18,998 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44087,1684727718327-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:55:18,998 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44087,1684727718327-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:55:18,998 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44087,1684727718327-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:55:18,998 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:44087, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:55:18,998 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-05-22 03:55:18,998 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-05-22 03:55:18,999 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-05-22 03:55:19,000 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-05-22 03:55:19,001 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-05-22 03:55:19,003 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-05-22 03:55:19,004 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-05-22 03:55:19,007 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/.tmp/data/hbase/namespace/47b192f2be256dfdce2a5a9b181dd5fd 2023-05-22 03:55:19,008 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/.tmp/data/hbase/namespace/47b192f2be256dfdce2a5a9b181dd5fd empty. 2023-05-22 03:55:19,008 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/.tmp/data/hbase/namespace/47b192f2be256dfdce2a5a9b181dd5fd 2023-05-22 03:55:19,008 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-05-22 03:55:19,018 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-05-22 03:55:19,019 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 47b192f2be256dfdce2a5a9b181dd5fd, NAME => 'hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/.tmp 2023-05-22 03:55:19,033 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:55:19,033 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 47b192f2be256dfdce2a5a9b181dd5fd, disabling compactions & flushes 2023-05-22 03:55:19,033 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:55:19,033 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:55:19,033 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. after waiting 0 ms 2023-05-22 03:55:19,033 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:55:19,033 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:55:19,033 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 47b192f2be256dfdce2a5a9b181dd5fd: 2023-05-22 03:55:19,036 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-05-22 03:55:19,037 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727719036"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1684727719036"}]},"ts":"1684727719036"} 2023-05-22 03:55:19,040 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-05-22 03:55:19,041 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-05-22 03:55:19,041 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727719041"}]},"ts":"1684727719041"} 2023-05-22 03:55:19,043 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-05-22 03:55:19,047 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=47b192f2be256dfdce2a5a9b181dd5fd, ASSIGN}] 2023-05-22 03:55:19,050 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=47b192f2be256dfdce2a5a9b181dd5fd, ASSIGN 2023-05-22 03:55:19,051 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=47b192f2be256dfdce2a5a9b181dd5fd, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,40121,1684727718365; forceNewPlan=false, retain=false 2023-05-22 03:55:19,202 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=47b192f2be256dfdce2a5a9b181dd5fd, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,40121,1684727718365 2023-05-22 03:55:19,203 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727719202"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1684727719202"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1684727719202"}]},"ts":"1684727719202"} 2023-05-22 03:55:19,205 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 47b192f2be256dfdce2a5a9b181dd5fd, server=jenkins-hbase20.apache.org,40121,1684727718365}] 2023-05-22 03:55:19,361 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:55:19,362 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 47b192f2be256dfdce2a5a9b181dd5fd, NAME => 'hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:55:19,362 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 47b192f2be256dfdce2a5a9b181dd5fd 2023-05-22 03:55:19,362 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:55:19,362 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 47b192f2be256dfdce2a5a9b181dd5fd 2023-05-22 03:55:19,362 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 47b192f2be256dfdce2a5a9b181dd5fd 2023-05-22 03:55:19,364 INFO [StoreOpener-47b192f2be256dfdce2a5a9b181dd5fd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 47b192f2be256dfdce2a5a9b181dd5fd 2023-05-22 03:55:19,365 DEBUG [StoreOpener-47b192f2be256dfdce2a5a9b181dd5fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/namespace/47b192f2be256dfdce2a5a9b181dd5fd/info 2023-05-22 03:55:19,365 DEBUG [StoreOpener-47b192f2be256dfdce2a5a9b181dd5fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/namespace/47b192f2be256dfdce2a5a9b181dd5fd/info 2023-05-22 03:55:19,366 INFO [StoreOpener-47b192f2be256dfdce2a5a9b181dd5fd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 47b192f2be256dfdce2a5a9b181dd5fd columnFamilyName info 2023-05-22 03:55:19,367 INFO [StoreOpener-47b192f2be256dfdce2a5a9b181dd5fd-1] regionserver.HStore(310): Store=47b192f2be256dfdce2a5a9b181dd5fd/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:55:19,368 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/namespace/47b192f2be256dfdce2a5a9b181dd5fd 2023-05-22 03:55:19,368 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/namespace/47b192f2be256dfdce2a5a9b181dd5fd 2023-05-22 03:55:19,372 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 47b192f2be256dfdce2a5a9b181dd5fd 2023-05-22 03:55:19,374 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/hbase/namespace/47b192f2be256dfdce2a5a9b181dd5fd/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:55:19,375 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 47b192f2be256dfdce2a5a9b181dd5fd; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=762376, jitterRate=-0.030589550733566284}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:55:19,375 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 47b192f2be256dfdce2a5a9b181dd5fd: 2023-05-22 03:55:19,378 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd., pid=6, masterSystemTime=1684727719357 2023-05-22 03:55:19,381 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:55:19,381 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:55:19,382 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=47b192f2be256dfdce2a5a9b181dd5fd, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,40121,1684727718365 2023-05-22 03:55:19,382 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727719382"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1684727719382"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1684727719382"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1684727719382"}]},"ts":"1684727719382"} 2023-05-22 03:55:19,387 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-05-22 03:55:19,387 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 47b192f2be256dfdce2a5a9b181dd5fd, server=jenkins-hbase20.apache.org,40121,1684727718365 in 179 msec 2023-05-22 03:55:19,389 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-05-22 03:55:19,390 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=47b192f2be256dfdce2a5a9b181dd5fd, ASSIGN in 340 msec 2023-05-22 03:55:19,390 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-05-22 03:55:19,391 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727719391"}]},"ts":"1684727719391"} 2023-05-22 03:55:19,393 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-05-22 03:55:19,399 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-05-22 03:55:19,401 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 400 msec 2023-05-22 03:55:19,402 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-05-22 03:55:19,413 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:55:19,413 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:55:19,418 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-05-22 03:55:19,429 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:55:19,434 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 16 msec 2023-05-22 03:55:19,441 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-05-22 03:55:19,448 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:55:19,451 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 10 msec 2023-05-22 03:55:19,465 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-05-22 03:55:19,466 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-05-22 03:55:19,466 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 1.062sec 2023-05-22 03:55:19,466 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-05-22 03:55:19,467 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-05-22 03:55:19,467 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-05-22 03:55:19,467 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44087,1684727718327-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-05-22 03:55:19,468 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44087,1684727718327-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-05-22 03:55:19,469 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-05-22 03:55:19,484 DEBUG [Listener at localhost.localdomain/35103] zookeeper.ReadOnlyZKClient(139): Connect 0x6c754f59 to 127.0.0.1:49349 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:55:19,489 DEBUG [Listener at localhost.localdomain/35103] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62397fed, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:55:19,491 DEBUG [hconnection-0x3b4badcf-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-05-22 03:55:19,494 INFO [RS-EventLoopGroup-9-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54674, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-05-22 03:55:19,496 INFO [Listener at localhost.localdomain/35103] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,44087,1684727718327 2023-05-22 03:55:19,496 INFO [Listener at localhost.localdomain/35103] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:55:19,500 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-05-22 03:55:19,500 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:55:19,501 INFO [Listener at localhost.localdomain/35103] master.MasterRpcServices(492): Client=null/null set balanceSwitch=false 2023-05-22 03:55:19,501 INFO [Listener at localhost.localdomain/35103] wal.TestLogRolling(429): Starting testLogRollOnPipelineRestart 2023-05-22 03:55:19,501 INFO [Listener at localhost.localdomain/35103] wal.TestLogRolling(432): Replication=2 2023-05-22 03:55:19,504 DEBUG [Listener at localhost.localdomain/35103] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-05-22 03:55:19,511 INFO [RS-EventLoopGroup-8-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:56906, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-05-22 03:55:19,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44087] util.TableDescriptorChecker(340): MAX_FILESIZE for table descriptor or "hbase.hregion.max.filesize" (786432) is too small, which might cause over splitting into unmanageable number of regions. 2023-05-22 03:55:19,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44087] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (8192) is too small, which might cause very frequent flushing. 2023-05-22 03:55:19,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44087] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestLogRolling-testLogRollOnPipelineRestart', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-05-22 03:55:19,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44087] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestLogRolling-testLogRollOnPipelineRestart 2023-05-22 03:55:19,519 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestLogRolling-testLogRollOnPipelineRestart execute state=CREATE_TABLE_PRE_OPERATION 2023-05-22 03:55:19,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44087] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestLogRolling-testLogRollOnPipelineRestart" procId is: 9 2023-05-22 03:55:19,520 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestLogRolling-testLogRollOnPipelineRestart execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-05-22 03:55:19,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44087] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-05-22 03:55:19,522 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/.tmp/data/default/TestLogRolling-testLogRollOnPipelineRestart/02d6639888ca39d8ac3cb793f09913bc 2023-05-22 03:55:19,523 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/.tmp/data/default/TestLogRolling-testLogRollOnPipelineRestart/02d6639888ca39d8ac3cb793f09913bc empty. 2023-05-22 03:55:19,523 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/.tmp/data/default/TestLogRolling-testLogRollOnPipelineRestart/02d6639888ca39d8ac3cb793f09913bc 2023-05-22 03:55:19,523 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestLogRolling-testLogRollOnPipelineRestart regions 2023-05-22 03:55:19,537 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/.tmp/data/default/TestLogRolling-testLogRollOnPipelineRestart/.tabledesc/.tableinfo.0000000001 2023-05-22 03:55:19,543 INFO [RegionOpenAndInit-TestLogRolling-testLogRollOnPipelineRestart-pool-0] regionserver.HRegion(7675): creating {ENCODED => 02d6639888ca39d8ac3cb793f09913bc, NAME => 'TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestLogRolling-testLogRollOnPipelineRestart', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/.tmp 2023-05-22 03:55:19,559 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRollOnPipelineRestart-pool-0] regionserver.HRegion(866): Instantiated TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:55:19,559 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRollOnPipelineRestart-pool-0] regionserver.HRegion(1604): Closing 02d6639888ca39d8ac3cb793f09913bc, disabling compactions & flushes 2023-05-22 03:55:19,559 INFO [RegionOpenAndInit-TestLogRolling-testLogRollOnPipelineRestart-pool-0] regionserver.HRegion(1626): Closing region TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:55:19,559 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRollOnPipelineRestart-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:55:19,559 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRollOnPipelineRestart-pool-0] regionserver.HRegion(1714): Acquired close lock on TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. after waiting 0 ms 2023-05-22 03:55:19,559 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRollOnPipelineRestart-pool-0] regionserver.HRegion(1724): Updates disabled for region TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:55:19,559 INFO [RegionOpenAndInit-TestLogRolling-testLogRollOnPipelineRestart-pool-0] regionserver.HRegion(1838): Closed TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:55:19,559 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRollOnPipelineRestart-pool-0] regionserver.HRegion(1558): Region close journal for 02d6639888ca39d8ac3cb793f09913bc: 2023-05-22 03:55:19,562 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestLogRolling-testLogRollOnPipelineRestart execute state=CREATE_TABLE_ADD_TO_META 2023-05-22 03:55:19,564 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc.","families":{"info":[{"qualifier":"regioninfo","vlen":77,"tag":[],"timestamp":"1684727719563"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1684727719563"}]},"ts":"1684727719563"} 2023-05-22 03:55:19,566 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-05-22 03:55:19,567 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestLogRolling-testLogRollOnPipelineRestart execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-05-22 03:55:19,568 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestLogRolling-testLogRollOnPipelineRestart","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727719567"}]},"ts":"1684727719567"} 2023-05-22 03:55:19,570 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestLogRolling-testLogRollOnPipelineRestart, state=ENABLING in hbase:meta 2023-05-22 03:55:19,573 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestLogRolling-testLogRollOnPipelineRestart, region=02d6639888ca39d8ac3cb793f09913bc, ASSIGN}] 2023-05-22 03:55:19,574 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestLogRolling-testLogRollOnPipelineRestart, region=02d6639888ca39d8ac3cb793f09913bc, ASSIGN 2023-05-22 03:55:19,575 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestLogRolling-testLogRollOnPipelineRestart, region=02d6639888ca39d8ac3cb793f09913bc, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,40121,1684727718365; forceNewPlan=false, retain=false 2023-05-22 03:55:19,727 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=02d6639888ca39d8ac3cb793f09913bc, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,40121,1684727718365 2023-05-22 03:55:19,727 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc.","families":{"info":[{"qualifier":"regioninfo","vlen":77,"tag":[],"timestamp":"1684727719726"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1684727719726"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1684727719726"}]},"ts":"1684727719726"} 2023-05-22 03:55:19,729 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 02d6639888ca39d8ac3cb793f09913bc, server=jenkins-hbase20.apache.org,40121,1684727718365}] 2023-05-22 03:55:19,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:55:19,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 02d6639888ca39d8ac3cb793f09913bc, NAME => 'TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:55:19,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestLogRolling-testLogRollOnPipelineRestart 02d6639888ca39d8ac3cb793f09913bc 2023-05-22 03:55:19,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:55:19,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 02d6639888ca39d8ac3cb793f09913bc 2023-05-22 03:55:19,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 02d6639888ca39d8ac3cb793f09913bc 2023-05-22 03:55:19,892 INFO [StoreOpener-02d6639888ca39d8ac3cb793f09913bc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 02d6639888ca39d8ac3cb793f09913bc 2023-05-22 03:55:19,900 DEBUG [StoreOpener-02d6639888ca39d8ac3cb793f09913bc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/default/TestLogRolling-testLogRollOnPipelineRestart/02d6639888ca39d8ac3cb793f09913bc/info 2023-05-22 03:55:19,900 DEBUG [StoreOpener-02d6639888ca39d8ac3cb793f09913bc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/default/TestLogRolling-testLogRollOnPipelineRestart/02d6639888ca39d8ac3cb793f09913bc/info 2023-05-22 03:55:19,901 INFO [StoreOpener-02d6639888ca39d8ac3cb793f09913bc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 02d6639888ca39d8ac3cb793f09913bc columnFamilyName info 2023-05-22 03:55:19,901 INFO [StoreOpener-02d6639888ca39d8ac3cb793f09913bc-1] regionserver.HStore(310): Store=02d6639888ca39d8ac3cb793f09913bc/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:55:19,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/default/TestLogRolling-testLogRollOnPipelineRestart/02d6639888ca39d8ac3cb793f09913bc 2023-05-22 03:55:19,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/default/TestLogRolling-testLogRollOnPipelineRestart/02d6639888ca39d8ac3cb793f09913bc 2023-05-22 03:55:19,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 02d6639888ca39d8ac3cb793f09913bc 2023-05-22 03:55:19,915 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/data/default/TestLogRolling-testLogRollOnPipelineRestart/02d6639888ca39d8ac3cb793f09913bc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:55:19,915 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 02d6639888ca39d8ac3cb793f09913bc; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=823488, jitterRate=0.0471201092004776}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:55:19,915 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 02d6639888ca39d8ac3cb793f09913bc: 2023-05-22 03:55:19,916 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc., pid=11, masterSystemTime=1684727719883 2023-05-22 03:55:19,918 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:55:19,918 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:55:19,918 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=02d6639888ca39d8ac3cb793f09913bc, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,40121,1684727718365 2023-05-22 03:55:19,919 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc.","families":{"info":[{"qualifier":"regioninfo","vlen":77,"tag":[],"timestamp":"1684727719918"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1684727719918"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1684727719918"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1684727719918"}]},"ts":"1684727719918"} 2023-05-22 03:55:19,923 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-05-22 03:55:19,923 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 02d6639888ca39d8ac3cb793f09913bc, server=jenkins-hbase20.apache.org,40121,1684727718365 in 192 msec 2023-05-22 03:55:19,925 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-05-22 03:55:19,925 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestLogRolling-testLogRollOnPipelineRestart, region=02d6639888ca39d8ac3cb793f09913bc, ASSIGN in 350 msec 2023-05-22 03:55:19,926 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestLogRolling-testLogRollOnPipelineRestart execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-05-22 03:55:19,926 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestLogRolling-testLogRollOnPipelineRestart","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727719926"}]},"ts":"1684727719926"} 2023-05-22 03:55:19,928 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestLogRolling-testLogRollOnPipelineRestart, state=ENABLED in hbase:meta 2023-05-22 03:55:19,930 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestLogRolling-testLogRollOnPipelineRestart execute state=CREATE_TABLE_POST_OPERATION 2023-05-22 03:55:19,932 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestLogRolling-testLogRollOnPipelineRestart in 415 msec 2023-05-22 03:55:22,361 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-05-22 03:55:24,625 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestLogRolling-testLogRollOnPipelineRestart' 2023-05-22 03:55:29,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44087] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-05-22 03:55:29,522 INFO [Listener at localhost.localdomain/35103] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestLogRolling-testLogRollOnPipelineRestart, procId: 9 completed 2023-05-22 03:55:29,525 DEBUG [Listener at localhost.localdomain/35103] hbase.HBaseTestingUtility(2627): Found 1 regions for table TestLogRolling-testLogRollOnPipelineRestart 2023-05-22 03:55:29,525 DEBUG [Listener at localhost.localdomain/35103] hbase.HBaseTestingUtility(2633): firstRegionName=TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:55:31,534 INFO [Listener at localhost.localdomain/35103] wal.TestLogRolling(469): log.getCurrentFileName()): hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727718758 2023-05-22 03:55:31,534 WARN [Listener at localhost.localdomain/35103] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:55:31,536 WARN [ResponseProcessor for block BP-69416852-148.251.75.209-1684727717821:blk_1073741829_1005] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-69416852-148.251.75.209-1684727717821:blk_1073741829_1005 java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:456) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1080) 2023-05-22 03:55:31,537 WARN [ResponseProcessor for block BP-69416852-148.251.75.209-1684727717821:blk_1073741833_1009] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-69416852-148.251.75.209-1684727717821:blk_1073741833_1009 java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:456) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1080) 2023-05-22 03:55:31,538 WARN [DataStreamer for file /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/WALs/jenkins-hbase20.apache.org,44087,1684727718327/jenkins-hbase20.apache.org%2C44087%2C1684727718327.1684727718485 block BP-69416852-148.251.75.209-1684727717821:blk_1073741829_1005] hdfs.DataStreamer(1548): Error Recovery for BP-69416852-148.251.75.209-1684727717821:blk_1073741829_1005 in pipeline [DatanodeInfoWithStorage[127.0.0.1:36747,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK], DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]]: datanode 0(DatanodeInfoWithStorage[127.0.0.1:36747,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK]) is bad. 2023-05-22 03:55:31,538 WARN [ResponseProcessor for block BP-69416852-148.251.75.209-1684727717821:blk_1073741832_1008] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-69416852-148.251.75.209-1684727717821:blk_1073741832_1008 java.io.IOException: Bad response ERROR for BP-69416852-148.251.75.209-1684727717821:blk_1073741832_1008 from datanode DatanodeInfoWithStorage[127.0.0.1:36747,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK] at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1120) 2023-05-22 03:55:31,538 WARN [DataStreamer for file /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.meta.1684727718938.meta block BP-69416852-148.251.75.209-1684727717821:blk_1073741833_1009] hdfs.DataStreamer(1548): Error Recovery for BP-69416852-148.251.75.209-1684727717821:blk_1073741833_1009 in pipeline [DatanodeInfoWithStorage[127.0.0.1:36747,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK], DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]]: datanode 0(DatanodeInfoWithStorage[127.0.0.1:36747,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK]) is bad. 2023-05-22 03:55:31,538 WARN [DataStreamer for file /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727718758 block BP-69416852-148.251.75.209-1684727717821:blk_1073741832_1008] hdfs.DataStreamer(1548): Error Recovery for BP-69416852-148.251.75.209-1684727717821:blk_1073741832_1008 in pipeline [DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK], DatanodeInfoWithStorage[127.0.0.1:36747,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK]]: datanode 1(DatanodeInfoWithStorage[127.0.0.1:36747,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK]) is bad. 2023-05-22 03:55:31,539 WARN [PacketResponder: BP-69416852-148.251.75.209-1684727717821:blk_1073741832_1008, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[127.0.0.1:36747]] datanode.BlockReceiver$PacketResponder(1486): IOException in BlockReceiver.run(): java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:477) at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1630) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1565) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1478) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:31,540 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_126213987_17 at /127.0.0.1:47892 [Receiving block BP-69416852-148.251.75.209-1684727717821:blk_1073741832_1008]] datanode.DataXceiver(323): 127.0.0.1:35699:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:47892 dst: /127.0.0.1:35699 java.io.IOException: Premature EOF from inputStream at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:31,544 INFO [Listener at localhost.localdomain/35103] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:55:31,546 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-1200990375_17 at /127.0.0.1:47874 [Receiving block BP-69416852-148.251.75.209-1684727717821:blk_1073741829_1005]] datanode.DataXceiver(323): 127.0.0.1:35699:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:47874 dst: /127.0.0.1:35699 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:35699 remote=/127.0.0.1:47874]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:31,547 WARN [PacketResponder: BP-69416852-148.251.75.209-1684727717821:blk_1073741833_1009, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[127.0.0.1:35699]] datanode.BlockReceiver$PacketResponder(1486): IOException in BlockReceiver.run(): java.io.IOException: The stream is closed at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1630) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1565) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1478) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:31,547 WARN [PacketResponder: BP-69416852-148.251.75.209-1684727717821:blk_1073741829_1005, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[127.0.0.1:35699]] datanode.BlockReceiver$PacketResponder(1486): IOException in BlockReceiver.run(): java.io.IOException: The stream is closed at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1630) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1565) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1478) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:31,546 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_126213987_17 at /127.0.0.1:47904 [Receiving block BP-69416852-148.251.75.209-1684727717821:blk_1073741833_1009]] datanode.DataXceiver(323): 127.0.0.1:35699:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:47904 dst: /127.0.0.1:35699 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:35699 remote=/127.0.0.1:47904]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:31,549 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-1200990375_17 at /127.0.0.1:53698 [Receiving block BP-69416852-148.251.75.209-1684727717821:blk_1073741829_1005]] datanode.DataXceiver(323): 127.0.0.1:36747:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:53698 dst: /127.0.0.1:36747 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:31,548 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_126213987_17 at /127.0.0.1:53746 [Receiving block BP-69416852-148.251.75.209-1684727717821:blk_1073741833_1009]] datanode.DataXceiver(323): 127.0.0.1:36747:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:53746 dst: /127.0.0.1:36747 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:31,648 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_126213987_17 at /127.0.0.1:53738 [Receiving block BP-69416852-148.251.75.209-1684727717821:blk_1073741832_1008]] datanode.DataXceiver(323): 127.0.0.1:36747:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:53738 dst: /127.0.0.1:36747 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:31,650 WARN [BP-69416852-148.251.75.209-1684727717821 heartbeating to localhost.localdomain/127.0.0.1:37871] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:55:31,650 WARN [BP-69416852-148.251.75.209-1684727717821 heartbeating to localhost.localdomain/127.0.0.1:37871] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-69416852-148.251.75.209-1684727717821 (Datanode Uuid 5181a363-6d97-47ef-86f3-4730977e7fb9) service to localhost.localdomain/127.0.0.1:37871 2023-05-22 03:55:31,650 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data3/current/BP-69416852-148.251.75.209-1684727717821] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:55:31,651 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data4/current/BP-69416852-148.251.75.209-1684727717821] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:55:31,658 WARN [Listener at localhost.localdomain/35103] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:55:31,662 WARN [Listener at localhost.localdomain/35103] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:55:31,664 INFO [Listener at localhost.localdomain/35103] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:55:31,672 INFO [Listener at localhost.localdomain/35103] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/java.io.tmpdir/Jetty_localhost_45827_datanode____.mdbh6l/webapp 2023-05-22 03:55:31,749 INFO [Listener at localhost.localdomain/35103] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45827 2023-05-22 03:55:31,756 WARN [Listener at localhost.localdomain/38795] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:55:31,761 WARN [Listener at localhost.localdomain/38795] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:55:31,761 WARN [ResponseProcessor for block BP-69416852-148.251.75.209-1684727717821:blk_1073741832_1015] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-69416852-148.251.75.209-1684727717821:blk_1073741832_1015 java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:456) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1080) 2023-05-22 03:55:31,761 WARN [ResponseProcessor for block BP-69416852-148.251.75.209-1684727717821:blk_1073741829_1014] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-69416852-148.251.75.209-1684727717821:blk_1073741829_1014 java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:456) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1080) 2023-05-22 03:55:31,761 WARN [ResponseProcessor for block BP-69416852-148.251.75.209-1684727717821:blk_1073741833_1016] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-69416852-148.251.75.209-1684727717821:blk_1073741833_1016 java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:456) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1080) 2023-05-22 03:55:31,765 INFO [Listener at localhost.localdomain/38795] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:55:31,815 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb9c5eedcbf3e3f48: Processing first storage report for DS-86ac7614-eb39-435c-9344-7c2530a2a13f from datanode 5181a363-6d97-47ef-86f3-4730977e7fb9 2023-05-22 03:55:31,815 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb9c5eedcbf3e3f48: from storage DS-86ac7614-eb39-435c-9344-7c2530a2a13f node DatanodeRegistration(127.0.0.1:42123, datanodeUuid=5181a363-6d97-47ef-86f3-4730977e7fb9, infoPort=40575, infoSecurePort=0, ipcPort=38795, storageInfo=lv=-57;cid=testClusterID;nsid=1219854411;c=1684727717821), blocks: 6, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:55:31,816 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb9c5eedcbf3e3f48: Processing first storage report for DS-b9e2f754-2dd9-4cee-aad4-2ae7b2c446a1 from datanode 5181a363-6d97-47ef-86f3-4730977e7fb9 2023-05-22 03:55:31,816 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb9c5eedcbf3e3f48: from storage DS-b9e2f754-2dd9-4cee-aad4-2ae7b2c446a1 node DatanodeRegistration(127.0.0.1:42123, datanodeUuid=5181a363-6d97-47ef-86f3-4730977e7fb9, infoPort=40575, infoSecurePort=0, ipcPort=38795, storageInfo=lv=-57;cid=testClusterID;nsid=1219854411;c=1684727717821), blocks: 7, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:55:31,869 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-1200990375_17 at /127.0.0.1:38208 [Receiving block BP-69416852-148.251.75.209-1684727717821:blk_1073741829_1005]] datanode.DataXceiver(323): 127.0.0.1:35699:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:38208 dst: /127.0.0.1:35699 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:31,872 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_126213987_17 at /127.0.0.1:38206 [Receiving block BP-69416852-148.251.75.209-1684727717821:blk_1073741833_1009]] datanode.DataXceiver(323): 127.0.0.1:35699:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:38206 dst: /127.0.0.1:35699 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:31,871 WARN [BP-69416852-148.251.75.209-1684727717821 heartbeating to localhost.localdomain/127.0.0.1:37871] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:55:31,869 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_126213987_17 at /127.0.0.1:38224 [Receiving block BP-69416852-148.251.75.209-1684727717821:blk_1073741832_1008]] datanode.DataXceiver(323): 127.0.0.1:35699:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:38224 dst: /127.0.0.1:35699 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:31,874 WARN [BP-69416852-148.251.75.209-1684727717821 heartbeating to localhost.localdomain/127.0.0.1:37871] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-69416852-148.251.75.209-1684727717821 (Datanode Uuid 6eb1cf74-c837-4445-a555-7e1f0435d520) service to localhost.localdomain/127.0.0.1:37871 2023-05-22 03:55:31,876 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data1/current/BP-69416852-148.251.75.209-1684727717821] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:55:31,877 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data2/current/BP-69416852-148.251.75.209-1684727717821] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:55:31,888 WARN [Listener at localhost.localdomain/38795] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:55:31,891 WARN [Listener at localhost.localdomain/38795] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:55:31,892 INFO [Listener at localhost.localdomain/38795] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:55:31,898 INFO [Listener at localhost.localdomain/38795] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/java.io.tmpdir/Jetty_localhost_34629_datanode____.ougg7x/webapp 2023-05-22 03:55:31,989 INFO [Listener at localhost.localdomain/38795] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34629 2023-05-22 03:55:31,997 WARN [Listener at localhost.localdomain/44247] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:55:32,061 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x57680323e0338abc: Processing first storage report for DS-085d43d7-a480-48a8-8078-32697106b7a8 from datanode 6eb1cf74-c837-4445-a555-7e1f0435d520 2023-05-22 03:55:32,061 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x57680323e0338abc: from storage DS-085d43d7-a480-48a8-8078-32697106b7a8 node DatanodeRegistration(127.0.0.1:43211, datanodeUuid=6eb1cf74-c837-4445-a555-7e1f0435d520, infoPort=44449, infoSecurePort=0, ipcPort=44247, storageInfo=lv=-57;cid=testClusterID;nsid=1219854411;c=1684727717821), blocks: 7, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:55:32,062 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x57680323e0338abc: Processing first storage report for DS-60858ccf-ea05-49f2-aeec-a20984865bec from datanode 6eb1cf74-c837-4445-a555-7e1f0435d520 2023-05-22 03:55:32,062 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x57680323e0338abc: from storage DS-60858ccf-ea05-49f2-aeec-a20984865bec node DatanodeRegistration(127.0.0.1:43211, datanodeUuid=6eb1cf74-c837-4445-a555-7e1f0435d520, infoPort=44449, infoSecurePort=0, ipcPort=44247, storageInfo=lv=-57;cid=testClusterID;nsid=1219854411;c=1684727717821), blocks: 6, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:55:33,005 INFO [Listener at localhost.localdomain/44247] wal.TestLogRolling(481): Data Nodes restarted 2023-05-22 03:55:33,010 INFO [Listener at localhost.localdomain/44247] wal.AbstractTestLogRolling(233): Validated row row1002 2023-05-22 03:55:33,011 WARN [RS:0;jenkins-hbase20:40121.append-pool-0] wal.FSHLog$RingBufferEventHandler(1203): Append sequenceId=5, requesting roll of WAL java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:33,013 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(197): WAL FSHLog jenkins-hbase20.apache.org%2C40121%2C1684727718365:(num 1684727718758) roll requested 2023-05-22 03:55:33,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40121] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=5, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1205) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1078) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:979) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.lang.Thread.run(Thread.java:750) Caused by: java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:33,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40121] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 1.2 K connection: 148.251.75.209:54674 deadline: 1684727743011, exception=org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=5, requesting roll of WAL 2023-05-22 03:55:33,025 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.TestLogRolling$7(456): preLogRoll: oldFile=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727718758 newFile=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727733013 2023-05-22 03:55:33,025 WARN [regionserver/jenkins-hbase20:0.logRoller] wal.FSHLog(374): Failed sync-before-close but no outstanding appends; closing WALorg.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=5, requesting roll of WAL 2023-05-22 03:55:33,026 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727718758 with entries=5, filesize=2.11 KB; new WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727733013 2023-05-22 03:55:33,026 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42123,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK], DatanodeInfoWithStorage[127.0.0.1:43211,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] 2023-05-22 03:55:33,026 WARN [Close-WAL-Writer-0] wal.AbstractProtobufLogWriter(237): Failed to write trailer, non-fatal, continuing... java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:33,026 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727718758 is not closed yet, will try archiving it next time 2023-05-22 03:55:33,026 WARN [Close-WAL-Writer-0] wal.FSHLog(466): Riding over failed WAL close of hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727718758; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:45,108 INFO [Listener at localhost.localdomain/44247] wal.AbstractTestLogRolling(233): Validated row row1003 2023-05-22 03:55:47,112 WARN [Listener at localhost.localdomain/44247] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:55:47,118 WARN [ResponseProcessor for block BP-69416852-148.251.75.209-1684727717821:blk_1073741838_1017] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-69416852-148.251.75.209-1684727717821:blk_1073741838_1017 java.io.IOException: Bad response ERROR for BP-69416852-148.251.75.209-1684727717821:blk_1073741838_1017 from datanode DatanodeInfoWithStorage[127.0.0.1:43211,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK] at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1120) 2023-05-22 03:55:47,118 WARN [DataStreamer for file /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727733013 block BP-69416852-148.251.75.209-1684727717821:blk_1073741838_1017] hdfs.DataStreamer(1548): Error Recovery for BP-69416852-148.251.75.209-1684727717821:blk_1073741838_1017 in pipeline [DatanodeInfoWithStorage[127.0.0.1:42123,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK], DatanodeInfoWithStorage[127.0.0.1:43211,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]]: datanode 1(DatanodeInfoWithStorage[127.0.0.1:43211,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]) is bad. 2023-05-22 03:55:47,119 WARN [PacketResponder: BP-69416852-148.251.75.209-1684727717821:blk_1073741838_1017, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[127.0.0.1:43211]] datanode.BlockReceiver$PacketResponder(1486): IOException in BlockReceiver.run(): java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:477) at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1630) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1565) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1478) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:47,120 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_126213987_17 at /127.0.0.1:51638 [Receiving block BP-69416852-148.251.75.209-1684727717821:blk_1073741838_1017]] datanode.DataXceiver(323): 127.0.0.1:42123:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:51638 dst: /127.0.0.1:42123 java.io.IOException: Premature EOF from inputStream at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:47,123 INFO [Listener at localhost.localdomain/44247] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:55:47,228 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_126213987_17 at /127.0.0.1:60746 [Receiving block BP-69416852-148.251.75.209-1684727717821:blk_1073741838_1017]] datanode.DataXceiver(323): 127.0.0.1:43211:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:60746 dst: /127.0.0.1:43211 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:47,230 WARN [BP-69416852-148.251.75.209-1684727717821 heartbeating to localhost.localdomain/127.0.0.1:37871] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:55:47,231 WARN [BP-69416852-148.251.75.209-1684727717821 heartbeating to localhost.localdomain/127.0.0.1:37871] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-69416852-148.251.75.209-1684727717821 (Datanode Uuid 6eb1cf74-c837-4445-a555-7e1f0435d520) service to localhost.localdomain/127.0.0.1:37871 2023-05-22 03:55:47,231 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data1/current/BP-69416852-148.251.75.209-1684727717821] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:55:47,232 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data2/current/BP-69416852-148.251.75.209-1684727717821] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:55:47,240 WARN [Listener at localhost.localdomain/44247] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:55:47,243 WARN [Listener at localhost.localdomain/44247] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:55:47,244 INFO [Listener at localhost.localdomain/44247] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:55:47,252 INFO [Listener at localhost.localdomain/44247] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/java.io.tmpdir/Jetty_localhost_38847_datanode____.frsusj/webapp 2023-05-22 03:55:47,330 INFO [Listener at localhost.localdomain/44247] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38847 2023-05-22 03:55:47,338 WARN [Listener at localhost.localdomain/34711] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:55:47,346 WARN [Listener at localhost.localdomain/34711] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:55:47,346 WARN [ResponseProcessor for block BP-69416852-148.251.75.209-1684727717821:blk_1073741838_1018] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-69416852-148.251.75.209-1684727717821:blk_1073741838_1018 java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:456) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1080) 2023-05-22 03:55:47,350 INFO [Listener at localhost.localdomain/34711] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:55:47,394 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8956eaa3f454dd99: Processing first storage report for DS-085d43d7-a480-48a8-8078-32697106b7a8 from datanode 6eb1cf74-c837-4445-a555-7e1f0435d520 2023-05-22 03:55:47,394 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8956eaa3f454dd99: from storage DS-085d43d7-a480-48a8-8078-32697106b7a8 node DatanodeRegistration(127.0.0.1:45611, datanodeUuid=6eb1cf74-c837-4445-a555-7e1f0435d520, infoPort=43759, infoSecurePort=0, ipcPort=34711, storageInfo=lv=-57;cid=testClusterID;nsid=1219854411;c=1684727717821), blocks: 8, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:55:47,395 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8956eaa3f454dd99: Processing first storage report for DS-60858ccf-ea05-49f2-aeec-a20984865bec from datanode 6eb1cf74-c837-4445-a555-7e1f0435d520 2023-05-22 03:55:47,395 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8956eaa3f454dd99: from storage DS-60858ccf-ea05-49f2-aeec-a20984865bec node DatanodeRegistration(127.0.0.1:45611, datanodeUuid=6eb1cf74-c837-4445-a555-7e1f0435d520, infoPort=43759, infoSecurePort=0, ipcPort=34711, storageInfo=lv=-57;cid=testClusterID;nsid=1219854411;c=1684727717821), blocks: 6, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:55:47,456 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_126213987_17 at /127.0.0.1:47806 [Receiving block BP-69416852-148.251.75.209-1684727717821:blk_1073741838_1017]] datanode.DataXceiver(323): 127.0.0.1:42123:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:47806 dst: /127.0.0.1:42123 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:55:47,461 WARN [BP-69416852-148.251.75.209-1684727717821 heartbeating to localhost.localdomain/127.0.0.1:37871] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:55:47,461 WARN [BP-69416852-148.251.75.209-1684727717821 heartbeating to localhost.localdomain/127.0.0.1:37871] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-69416852-148.251.75.209-1684727717821 (Datanode Uuid 5181a363-6d97-47ef-86f3-4730977e7fb9) service to localhost.localdomain/127.0.0.1:37871 2023-05-22 03:55:47,462 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data3/current/BP-69416852-148.251.75.209-1684727717821] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:55:47,462 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data4/current/BP-69416852-148.251.75.209-1684727717821] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:55:47,472 WARN [Listener at localhost.localdomain/34711] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:55:47,474 WARN [Listener at localhost.localdomain/34711] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:55:47,475 INFO [Listener at localhost.localdomain/34711] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:55:47,481 INFO [Listener at localhost.localdomain/34711] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/java.io.tmpdir/Jetty_localhost_33097_datanode____.1q40ff/webapp 2023-05-22 03:55:47,553 INFO [Listener at localhost.localdomain/34711] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33097 2023-05-22 03:55:47,560 WARN [Listener at localhost.localdomain/39159] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:55:47,613 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8d5afe3b8175385e: Processing first storage report for DS-86ac7614-eb39-435c-9344-7c2530a2a13f from datanode 5181a363-6d97-47ef-86f3-4730977e7fb9 2023-05-22 03:55:47,614 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8d5afe3b8175385e: from storage DS-86ac7614-eb39-435c-9344-7c2530a2a13f node DatanodeRegistration(127.0.0.1:34561, datanodeUuid=5181a363-6d97-47ef-86f3-4730977e7fb9, infoPort=43383, infoSecurePort=0, ipcPort=39159, storageInfo=lv=-57;cid=testClusterID;nsid=1219854411;c=1684727717821), blocks: 6, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:55:47,614 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8d5afe3b8175385e: Processing first storage report for DS-b9e2f754-2dd9-4cee-aad4-2ae7b2c446a1 from datanode 5181a363-6d97-47ef-86f3-4730977e7fb9 2023-05-22 03:55:47,614 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8d5afe3b8175385e: from storage DS-b9e2f754-2dd9-4cee-aad4-2ae7b2c446a1 node DatanodeRegistration(127.0.0.1:34561, datanodeUuid=5181a363-6d97-47ef-86f3-4730977e7fb9, infoPort=43383, infoSecurePort=0, ipcPort=39159, storageInfo=lv=-57;cid=testClusterID;nsid=1219854411;c=1684727717821), blocks: 8, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:55:48,560 WARN [master/jenkins-hbase20:0:becomeActiveMaster.append-pool-0] wal.FSHLog$RingBufferEventHandler(1203): Append sequenceId=91, requesting roll of WAL java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:48,563 DEBUG [master:store-WAL-Roller] wal.AbstractWALRoller(197): WAL FSHLog jenkins-hbase20.apache.org%2C44087%2C1684727718327:(num 1684727718485) roll requested 2023-05-22 03:55:48,563 ERROR [ProcExecTimeout] helpers.MarkerIgnoringBase(151): Failed to delete pids=[4, 7, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=91, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1205) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1078) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:979) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.lang.Thread.run(Thread.java:750) Caused by: java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:48,564 ERROR [ProcExecTimeout] procedure2.TimeoutExecutorThread(124): Ignoring pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner exception: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=91, requesting roll of WAL java.io.UncheckedIOException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=91, requesting roll of WAL at org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore.delete(RegionProcedureStore.java:423) at org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner.periodicExecute(CompletedProcedureCleaner.java:135) at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.executeInMemoryChore(TimeoutExecutorThread.java:122) at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.execDelayedProcedure(TimeoutExecutorThread.java:101) at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:68) Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=91, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1205) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1078) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:979) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.lang.Thread.run(Thread.java:750) Caused by: java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:48,566 INFO [Listener at localhost.localdomain/39159] wal.TestLogRolling(498): Data Nodes restarted 2023-05-22 03:55:48,568 INFO [Listener at localhost.localdomain/39159] wal.AbstractTestLogRolling(233): Validated row row1004 2023-05-22 03:55:48,570 WARN [RS:0;jenkins-hbase20:40121.append-pool-0] wal.FSHLog$RingBufferEventHandler(1203): Append sequenceId=8, requesting roll of WAL java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:42123,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:48,570 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(197): WAL FSHLog jenkins-hbase20.apache.org%2C40121%2C1684727718365:(num 1684727733013) roll requested 2023-05-22 03:55:48,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40121] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=8, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1205) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1078) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:979) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.lang.Thread.run(Thread.java:750) Caused by: java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:42123,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:48,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40121] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 1.2 K connection: 148.251.75.209:54674 deadline: 1684727758569, exception=org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=8, requesting roll of WAL 2023-05-22 03:55:48,577 WARN [master:store-WAL-Roller] wal.FSHLog(374): Failed sync-before-close but no outstanding appends; closing WALorg.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=91, requesting roll of WAL 2023-05-22 03:55:48,577 INFO [master:store-WAL-Roller] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/WALs/jenkins-hbase20.apache.org,44087,1684727718327/jenkins-hbase20.apache.org%2C44087%2C1684727718327.1684727718485 with entries=88, filesize=43.82 KB; new WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/WALs/jenkins-hbase20.apache.org,44087,1684727718327/jenkins-hbase20.apache.org%2C44087%2C1684727718327.1684727748563 2023-05-22 03:55:48,577 DEBUG [master:store-WAL-Roller] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45611,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK], DatanodeInfoWithStorage[127.0.0.1:34561,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK]] 2023-05-22 03:55:48,577 DEBUG [master:store-WAL-Roller] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/WALs/jenkins-hbase20.apache.org,44087,1684727718327/jenkins-hbase20.apache.org%2C44087%2C1684727718327.1684727718485 is not closed yet, will try archiving it next time 2023-05-22 03:55:48,577 WARN [Close-WAL-Writer-0] wal.AbstractProtobufLogWriter(237): Failed to write trailer, non-fatal, continuing... java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:48,578 WARN [Close-WAL-Writer-0] wal.FSHLog(466): Riding over failed WAL close of hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/WALs/jenkins-hbase20.apache.org,44087,1684727718327/jenkins-hbase20.apache.org%2C44087%2C1684727718327.1684727718485; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:48,586 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.TestLogRolling$7(456): preLogRoll: oldFile=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727733013 newFile=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727748570 2023-05-22 03:55:48,586 WARN [regionserver/jenkins-hbase20:0.logRoller] wal.FSHLog(374): Failed sync-before-close but no outstanding appends; closing WALorg.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=8, requesting roll of WAL 2023-05-22 03:55:48,586 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727733013 with entries=2, filesize=2.37 KB; new WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727748570 2023-05-22 03:55:48,586 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34561,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK], DatanodeInfoWithStorage[127.0.0.1:45611,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] 2023-05-22 03:55:48,586 WARN [Close-WAL-Writer-0] wal.AbstractProtobufLogWriter(237): Failed to write trailer, non-fatal, continuing... java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:42123,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:55:48,586 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727733013 is not closed yet, will try archiving it next time 2023-05-22 03:55:48,586 WARN [Close-WAL-Writer-0] wal.FSHLog(466): Riding over failed WAL close of hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727733013; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:42123,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:56:00,637 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling$7(456): preLogRoll: oldFile=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727748570 newFile=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628 2023-05-22 03:56:00,639 INFO [Listener at localhost.localdomain/39159] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727748570 with entries=1, filesize=1.22 KB; new WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628 2023-05-22 03:56:00,643 DEBUG [Listener at localhost.localdomain/39159] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34561,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK], DatanodeInfoWithStorage[127.0.0.1:45611,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] 2023-05-22 03:56:00,643 DEBUG [Listener at localhost.localdomain/39159] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727748570 is not closed yet, will try archiving it next time 2023-05-22 03:56:00,643 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling(512): recovering lease for hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727718758 2023-05-22 03:56:00,644 INFO [Listener at localhost.localdomain/39159] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727718758 2023-05-22 03:56:00,646 WARN [IPC Server handler 3 on default port 37871] namenode.FSNamesystem(3291): DIR* NameSystem.internalReleaseLease: File /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727718758 has not been closed. Lease recovery is in progress. RecoveryId = 1022 for block blk_1073741832_1015 2023-05-22 03:56:00,648 INFO [Listener at localhost.localdomain/39159] util.RecoverLeaseFSUtils(175): Failed to recover lease, attempt=0 on file=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727718758 after 4ms 2023-05-22 03:56:01,640 WARN [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@20a5f3db] datanode.BlockRecoveryWorker$RecoveryTaskContiguous(155): Failed to recover block (block=BP-69416852-148.251.75.209-1684727717821:blk_1073741832_1015, datanode=DatanodeInfoWithStorage[127.0.0.1:34561,null,null]) java.io.IOException: replica.getGenerationStamp() < block.getGenerationStamp(), block=blk_1073741832_1015, replica=ReplicaWaitingToBeRecovered, blk_1073741832_1008, RWR getNumBytes() = 2162 getBytesOnDisk() = 2162 getVisibleLength()= -1 getVolume() = /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data4/current getBlockFile() = /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data4/current/BP-69416852-148.251.75.209-1684727717821/current/rbw/blk_1073741832 at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecoveryImpl(FsDatasetImpl.java:2694) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecovery(FsDatasetImpl.java:2655) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecovery(FsDatasetImpl.java:2644) at org.apache.hadoop.hdfs.server.datanode.DataNode.initReplicaRecovery(DataNode.java:2835) at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker.callInitReplicaRecovery(BlockRecoveryWorker.java:346) at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker.access$300(BlockRecoveryWorker.java:46) at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskContiguous.recover(BlockRecoveryWorker.java:120) at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1.run(BlockRecoveryWorker.java:383) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:56:04,649 INFO [Listener at localhost.localdomain/39159] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=1 on file=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727718758 after 4005ms 2023-05-22 03:56:04,649 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling(516): Reading WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727718758 2023-05-22 03:56:04,663 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling(522): #3: [\x00/METAFAMILY:HBASE::REGION_EVENT::REGION_OPEN/1684727719375/Put/vlen=176/seqid=0] 2023-05-22 03:56:04,663 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling(522): #4: [default/info:d/1684727719425/Put/vlen=9/seqid=0] 2023-05-22 03:56:04,664 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling(522): #5: [hbase/info:d/1684727719445/Put/vlen=7/seqid=0] 2023-05-22 03:56:04,664 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling(522): #3: [\x00/METAFAMILY:HBASE::REGION_EVENT::REGION_OPEN/1684727719915/Put/vlen=232/seqid=0] 2023-05-22 03:56:04,664 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling(522): #4: [row1002/info:/1684727729532/Put/vlen=1045/seqid=0] 2023-05-22 03:56:04,664 DEBUG [Listener at localhost.localdomain/39159] wal.ProtobufLogReader(420): EOF at position 2162 2023-05-22 03:56:04,664 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling(512): recovering lease for hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727733013 2023-05-22 03:56:04,665 INFO [Listener at localhost.localdomain/39159] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727733013 2023-05-22 03:56:04,665 WARN [IPC Server handler 1 on default port 37871] namenode.FSNamesystem(3291): DIR* NameSystem.internalReleaseLease: File /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727733013 has not been closed. Lease recovery is in progress. RecoveryId = 1023 for block blk_1073741838_1018 2023-05-22 03:56:04,666 INFO [Listener at localhost.localdomain/39159] util.RecoverLeaseFSUtils(175): Failed to recover lease, attempt=0 on file=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727733013 after 1ms 2023-05-22 03:56:05,622 WARN [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@1210736a] datanode.BlockRecoveryWorker$RecoveryTaskContiguous(155): Failed to recover block (block=BP-69416852-148.251.75.209-1684727717821:blk_1073741838_1018, datanode=DatanodeInfoWithStorage[127.0.0.1:45611,null,null]) java.io.IOException: replica.getGenerationStamp() < block.getGenerationStamp(), block=blk_1073741838_1018, replica=ReplicaWaitingToBeRecovered, blk_1073741838_1017, RWR getNumBytes() = 2425 getBytesOnDisk() = 2425 getVisibleLength()= -1 getVolume() = /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data1/current getBlockFile() = /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data1/current/BP-69416852-148.251.75.209-1684727717821/current/rbw/blk_1073741838 at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecoveryImpl(FsDatasetImpl.java:2694) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecovery(FsDatasetImpl.java:2655) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecovery(FsDatasetImpl.java:2644) at org.apache.hadoop.hdfs.server.datanode.DataNode.initReplicaRecovery(DataNode.java:2835) at org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolServerSideTranslatorPB.initReplicaRecovery(InterDatanodeProtocolServerSideTranslatorPB.java:55) at org.apache.hadoop.hdfs.protocol.proto.InterDatanodeProtocolProtos$InterDatanodeProtocolService$2.callBlockingMethod(InterDatanodeProtocolProtos.java:3105) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2833) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110) at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker.callInitReplicaRecovery(BlockRecoveryWorker.java:348) at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker.access$300(BlockRecoveryWorker.java:46) at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskContiguous.recover(BlockRecoveryWorker.java:120) at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1.run(BlockRecoveryWorker.java:383) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): replica.getGenerationStamp() < block.getGenerationStamp(), block=blk_1073741838_1018, replica=ReplicaWaitingToBeRecovered, blk_1073741838_1017, RWR getNumBytes() = 2425 getBytesOnDisk() = 2425 getVisibleLength()= -1 getVolume() = /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data1/current getBlockFile() = /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data1/current/BP-69416852-148.251.75.209-1684727717821/current/rbw/blk_1073741838 at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecoveryImpl(FsDatasetImpl.java:2694) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecovery(FsDatasetImpl.java:2655) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecovery(FsDatasetImpl.java:2644) at org.apache.hadoop.hdfs.server.datanode.DataNode.initReplicaRecovery(DataNode.java:2835) at org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolServerSideTranslatorPB.initReplicaRecovery(InterDatanodeProtocolServerSideTranslatorPB.java:55) at org.apache.hadoop.hdfs.protocol.proto.InterDatanodeProtocolProtos$InterDatanodeProtocolService$2.callBlockingMethod(InterDatanodeProtocolProtos.java:3105) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2833) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1540) at org.apache.hadoop.ipc.Client.call(Client.java:1486) at org.apache.hadoop.ipc.Client.call(Client.java:1385) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy43.initReplicaRecovery(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolTranslatorPB.initReplicaRecovery(InterDatanodeProtocolTranslatorPB.java:83) at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker.callInitReplicaRecovery(BlockRecoveryWorker.java:346) ... 4 more 2023-05-22 03:56:08,667 INFO [Listener at localhost.localdomain/39159] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=1 on file=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727733013 after 4002ms 2023-05-22 03:56:08,668 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling(516): Reading WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727733013 2023-05-22 03:56:08,676 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling(522): #6: [row1003/info:/1684727743101/Put/vlen=1045/seqid=0] 2023-05-22 03:56:08,676 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling(522): #7: [row1004/info:/1684727745109/Put/vlen=1045/seqid=0] 2023-05-22 03:56:08,676 DEBUG [Listener at localhost.localdomain/39159] wal.ProtobufLogReader(420): EOF at position 2425 2023-05-22 03:56:08,676 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling(512): recovering lease for hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727748570 2023-05-22 03:56:08,676 INFO [Listener at localhost.localdomain/39159] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727748570 2023-05-22 03:56:08,677 INFO [Listener at localhost.localdomain/39159] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727748570 after 1ms 2023-05-22 03:56:08,677 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling(516): Reading WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727748570 2023-05-22 03:56:08,683 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling(522): #9: [row1005/info:/1684727758625/Put/vlen=1045/seqid=0] 2023-05-22 03:56:08,683 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling(512): recovering lease for hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628 2023-05-22 03:56:08,683 INFO [Listener at localhost.localdomain/39159] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628 2023-05-22 03:56:08,684 WARN [IPC Server handler 0 on default port 37871] namenode.FSNamesystem(3291): DIR* NameSystem.internalReleaseLease: File /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628 has not been closed. Lease recovery is in progress. RecoveryId = 1024 for block blk_1073741841_1021 2023-05-22 03:56:08,684 INFO [Listener at localhost.localdomain/39159] util.RecoverLeaseFSUtils(175): Failed to recover lease, attempt=0 on file=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628 after 1ms 2023-05-22 03:56:09,617 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-1200990375_17 at /127.0.0.1:32834 [Receiving block BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021]] datanode.DataXceiver(323): 127.0.0.1:34561:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:32834 dst: /127.0.0.1:34561 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:34561 remote=/127.0.0.1:32834]. 60000 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:56:09,619 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_-1200990375_17 at /127.0.0.1:36054 [Receiving block BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021]] datanode.DataXceiver(323): 127.0.0.1:45611:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:36054 dst: /127.0.0.1:45611 java.io.IOException: Premature EOF from inputStream at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:56:09,619 WARN [ResponseProcessor for block BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021] hdfs.DataStreamer$ResponseProcessor(1190): Exception for BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021 java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:456) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1080) 2023-05-22 03:56:09,621 WARN [DataStreamer for file /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628 block BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021] hdfs.DataStreamer(1548): Error Recovery for BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021 in pipeline [DatanodeInfoWithStorage[127.0.0.1:34561,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK], DatanodeInfoWithStorage[127.0.0.1:45611,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]]: datanode 0(DatanodeInfoWithStorage[127.0.0.1:34561,DS-86ac7614-eb39-435c-9344-7c2530a2a13f,DISK]) is bad. 2023-05-22 03:56:09,631 WARN [DataStreamer for file /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628 block BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021] hdfs.DataStreamer(823): DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unexpected BlockUCState: BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021 is UNDER_RECOVERY but not UNDER_CONSTRUCTION at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:4886) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:4955) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:954) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:992) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2833) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1540) at org.apache.hadoop.ipc.Client.call(Client.java:1486) at org.apache.hadoop.ipc.Client.call(Client.java:1385) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy30.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:918) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy33.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.DataStreamer.updateBlockForPipeline(DataStreamer.java:1602) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1479) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:56:12,686 INFO [Listener at localhost.localdomain/39159] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=1 on file=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628 after 4003ms 2023-05-22 03:56:12,686 DEBUG [Listener at localhost.localdomain/39159] wal.TestLogRolling(516): Reading WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628 2023-05-22 03:56:12,696 DEBUG [Listener at localhost.localdomain/39159] wal.ProtobufLogReader(420): EOF at position 83 2023-05-22 03:56:12,697 INFO [Listener at localhost.localdomain/39159] regionserver.HRegion(2745): Flushing 47b192f2be256dfdce2a5a9b181dd5fd 1/1 column families, dataSize=78 B heapSize=488 B 2023-05-22 03:56:12,698 WARN [RS:0;jenkins-hbase20:40121.append-pool-0] wal.FSHLog$RingBufferEventHandler(1203): Append sequenceId=7, requesting roll of WAL org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unexpected BlockUCState: BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021 is UNDER_RECOVERY but not UNDER_CONSTRUCTION at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:4886) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:4955) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:954) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:992) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2833) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1540) at org.apache.hadoop.ipc.Client.call(Client.java:1486) at org.apache.hadoop.ipc.Client.call(Client.java:1385) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy30.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:918) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy33.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.DataStreamer.updateBlockForPipeline(DataStreamer.java:1602) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1479) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:56:12,698 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(197): WAL FSHLog jenkins-hbase20.apache.org%2C40121%2C1684727718365:(num 1684727760628) roll requested 2023-05-22 03:56:12,698 DEBUG [Listener at localhost.localdomain/39159] regionserver.HRegion(2446): Flush status journal for 47b192f2be256dfdce2a5a9b181dd5fd: 2023-05-22 03:56:12,698 INFO [Listener at localhost.localdomain/39159] wal.TestLogRolling(551): org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=7, requesting roll of WAL org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=7, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1205) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1078) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:979) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unexpected BlockUCState: BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021 is UNDER_RECOVERY but not UNDER_CONSTRUCTION at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:4886) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:4955) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:954) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:992) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2833) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1540) at org.apache.hadoop.ipc.Client.call(Client.java:1486) at org.apache.hadoop.ipc.Client.call(Client.java:1385) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy30.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:918) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy33.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.DataStreamer.updateBlockForPipeline(DataStreamer.java:1602) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1479) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:56:12,700 INFO [Listener at localhost.localdomain/39159] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=2.96 KB heapSize=5.48 KB 2023-05-22 03:56:12,701 WARN [RS_OPEN_META-regionserver/jenkins-hbase20:0-0.append-pool-0] wal.FSHLog$RingBufferEventHandler(1203): Append sequenceId=15, requesting roll of WAL java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:56:12,702 DEBUG [Listener at localhost.localdomain/39159] regionserver.HRegion(2446): Flush status journal for 1588230740: 2023-05-22 03:56:12,702 INFO [Listener at localhost.localdomain/39159] wal.TestLogRolling(551): org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=15, requesting roll of WAL org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=15, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1205) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1078) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:979) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.lang.Thread.run(Thread.java:750) Caused by: java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:56:12,704 INFO [Listener at localhost.localdomain/39159] regionserver.HRegion(2745): Flushing 02d6639888ca39d8ac3cb793f09913bc 1/1 column families, dataSize=4.20 KB heapSize=4.75 KB 2023-05-22 03:56:12,704 DEBUG [Listener at localhost.localdomain/39159] regionserver.HRegion(2446): Flush status journal for 02d6639888ca39d8ac3cb793f09913bc: 2023-05-22 03:56:12,704 INFO [Listener at localhost.localdomain/39159] wal.TestLogRolling(551): org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=7, requesting roll of WAL org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=7, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1205) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1078) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:979) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unexpected BlockUCState: BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021 is UNDER_RECOVERY but not UNDER_CONSTRUCTION at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:4886) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:4955) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:954) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:992) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2833) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1540) at org.apache.hadoop.ipc.Client.call(Client.java:1486) at org.apache.hadoop.ipc.Client.call(Client.java:1385) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy30.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:918) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy33.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.DataStreamer.updateBlockForPipeline(DataStreamer.java:1602) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1479) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:56:12,712 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-05-22 03:56:12,712 INFO [Listener at localhost.localdomain/39159] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-05-22 03:56:12,717 DEBUG [Listener at localhost.localdomain/39159] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c754f59 to 127.0.0.1:49349 2023-05-22 03:56:12,717 DEBUG [Listener at localhost.localdomain/39159] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:56:12,717 DEBUG [Listener at localhost.localdomain/39159] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-05-22 03:56:12,717 DEBUG [Listener at localhost.localdomain/39159] util.JVMClusterUtil(257): Found active master hash=930703574, stopped=false 2023-05-22 03:56:12,717 INFO [Listener at localhost.localdomain/39159] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,44087,1684727718327 2023-05-22 03:56:12,719 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.TestLogRolling$7(456): preLogRoll: oldFile=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628 newFile=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727772698 2023-05-22 03:56:12,719 WARN [regionserver/jenkins-hbase20:0.logRoller] wal.FSHLog(374): Failed sync-before-close but no outstanding appends; closing WALorg.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=7, requesting roll of WAL 2023-05-22 03:56:12,719 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628 with entries=0, filesize=83 B; new WAL /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727772698 2023-05-22 03:56:12,719 WARN [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractProtobufLogWriter(237): Failed to write trailer, non-fatal, continuing... org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unexpected BlockUCState: BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021 is UNDER_RECOVERY but not UNDER_CONSTRUCTION at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:4886) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:4955) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:954) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:992) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2833) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1540) at org.apache.hadoop.ipc.Client.call(Client.java:1486) at org.apache.hadoop.ipc.Client.call(Client.java:1385) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy30.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:918) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy33.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.DataStreamer.updateBlockForPipeline(DataStreamer.java:1602) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1479) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:56:12,719 ERROR [regionserver/jenkins-hbase20:0.logRoller] wal.FSHLog(462): Close of WAL hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628 failed. Cause="Unexpected BlockUCState: BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021 is UNDER_RECOVERY but not UNDER_CONSTRUCTION at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:4886) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:4955) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:954) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:992) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2833) ", errors=3, hasUnflushedEntries=false 2023-05-22 03:56:12,719 ERROR [regionserver/jenkins-hbase20:0.logRoller] wal.FSHLog(426): Failed close of WAL writer hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628, unflushedEntries=0 org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unexpected BlockUCState: BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021 is UNDER_RECOVERY but not UNDER_CONSTRUCTION at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:4886) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:4955) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:954) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:992) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2833) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1540) at org.apache.hadoop.ipc.Client.call(Client.java:1486) at org.apache.hadoop.ipc.Client.call(Client.java:1385) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy30.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:918) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy33.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.DataStreamer.updateBlockForPipeline(DataStreamer.java:1602) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1479) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:56:12,720 ERROR [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(221): Roll wal failed and waiting timeout, will not retry org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException: hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628, unflushedEntries=0 at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doReplaceWriter(FSHLog.java:427) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doReplaceWriter(FSHLog.java:70) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.replaceWriter(AbstractFSWAL.java:828) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:884) at org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:304) at org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:211) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unexpected BlockUCState: BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021 is UNDER_RECOVERY but not UNDER_CONSTRUCTION at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:4886) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:4955) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:954) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:992) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2833) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1540) at org.apache.hadoop.ipc.Client.call(Client.java:1486) at org.apache.hadoop.ipc.Client.call(Client.java:1385) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy30.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:918) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy33.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.DataStreamer.updateBlockForPipeline(DataStreamer.java:1602) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1479) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:56:12,720 INFO [Listener at localhost.localdomain/39159] procedure2.ProcedureExecutor(629): Stopping 2023-05-22 03:56:12,720 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): regionserver:40121-0x101714ef9d00001, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-05-22 03:56:12,721 DEBUG [Listener at localhost.localdomain/39159] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6eec54ca to 127.0.0.1:49349 2023-05-22 03:56:12,721 DEBUG [Listener at localhost.localdomain/39159] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:56:12,720 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-05-22 03:56:12,722 INFO [Listener at localhost.localdomain/39159] regionserver.HRegionServer(2295): ***** STOPPING region server 'jenkins-hbase20.apache.org,40121,1684727718365' ***** 2023-05-22 03:56:12,722 INFO [Listener at localhost.localdomain/39159] regionserver.HRegionServer(2309): STOPPED: Shutdown requested 2023-05-22 03:56:12,722 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:56:12,721 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365 2023-05-22 03:56:12,723 INFO [RS:0;jenkins-hbase20:40121] regionserver.HeapMemoryManager(220): Stopping 2023-05-22 03:56:12,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40121-0x101714ef9d00001, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:56:12,723 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-05-22 03:56:12,723 INFO [RS:0;jenkins-hbase20:40121] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-05-22 03:56:12,723 WARN [WAL-Shutdown-0] wal.AbstractProtobufLogWriter(237): Failed to write trailer, non-fatal, continuing... java.nio.channels.ClosedChannelException at org.apache.hadoop.hdfs.DataStreamer$LastExceptionInStreamer.throwException4Close(DataStreamer.java:324) at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:151) at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:105) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58) at java.io.DataOutputStream.write(DataOutputStream.java:107) at java.io.FilterOutputStream.write(FilterOutputStream.java:97) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.writeWALTrailerAndMagic(ProtobufLogWriter.java:140) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:234) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.close(ProtobufLogWriter.java:67) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doShutdown(FSHLog.java:492) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL$2.call(AbstractFSWAL.java:951) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL$2.call(AbstractFSWAL.java:946) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-05-22 03:56:12,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:56:12,724 INFO [RS:0;jenkins-hbase20:40121] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-05-22 03:56:12,724 INFO [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(3303): Received CLOSE for 47b192f2be256dfdce2a5a9b181dd5fd 2023-05-22 03:56:12,724 INFO [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(3303): Received CLOSE for 02d6639888ca39d8ac3cb793f09913bc 2023-05-22 03:56:12,724 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365 2023-05-22 03:56:12,724 INFO [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,40121,1684727718365 2023-05-22 03:56:12,726 WARN [WAL-Shutdown-0] wal.AbstractProtobufLogWriter(237): Failed to write trailer, non-fatal, continuing... java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:56:12,724 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 47b192f2be256dfdce2a5a9b181dd5fd, disabling compactions & flushes 2023-05-22 03:56:12,726 DEBUG [RS:0;jenkins-hbase20:40121] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3bdc7e65 to 127.0.0.1:49349 2023-05-22 03:56:12,726 WARN [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(165): Failed to shutdown wal java.io.IOException: All datanodes [DatanodeInfoWithStorage[127.0.0.1:35699,DS-085d43d7-a480-48a8-8078-32697106b7a8,DISK]] are bad. Aborting... at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1537) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1472) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:56:12,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:56:12,726 ERROR [regionserver/jenkins-hbase20:0.logRoller] helpers.MarkerIgnoringBase(159): ***** ABORTING region server jenkins-hbase20.apache.org,40121,1684727718365: Failed log close in log roller ***** org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException: hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628, unflushedEntries=0 at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doReplaceWriter(FSHLog.java:427) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doReplaceWriter(FSHLog.java:70) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.replaceWriter(AbstractFSWAL.java:828) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:884) at org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:304) at org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:211) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unexpected BlockUCState: BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021 is UNDER_RECOVERY but not UNDER_CONSTRUCTION at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:4886) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:4955) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:954) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:992) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2833) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1540) at org.apache.hadoop.ipc.Client.call(Client.java:1486) at org.apache.hadoop.ipc.Client.call(Client.java:1385) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy30.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:918) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy33.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.DataStreamer.updateBlockForPipeline(DataStreamer.java:1602) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1479) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:56:12,726 DEBUG [RS:0;jenkins-hbase20:40121] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:56:12,726 ERROR [regionserver/jenkins-hbase20:0.logRoller] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-05-22 03:56:12,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:56:12,726 INFO [RS:0;jenkins-hbase20:40121] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-05-22 03:56:12,727 INFO [RS:0;jenkins-hbase20:40121] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-05-22 03:56:12,727 INFO [RS:0;jenkins-hbase20:40121] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-05-22 03:56:12,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. after waiting 0 ms 2023-05-22 03:56:12,727 INFO [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(3303): Received CLOSE for 1588230740 2023-05-22 03:56:12,726 DEBUG [regionserver/jenkins-hbase20:0.logRoller] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-05-22 03:56:12,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:56:12,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 47b192f2be256dfdce2a5a9b181dd5fd: 2023-05-22 03:56:12,727 INFO [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-05-22 03:56:12,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2539): Abort already in progress. Ignoring the current request with reason: Unrecoverable exception while closing hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:56:12,727 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-05-22 03:56:12,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 02d6639888ca39d8ac3cb793f09913bc, disabling compactions & flushes 2023-05-22 03:56:12,727 DEBUG [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(1478): Online Regions={47b192f2be256dfdce2a5a9b181dd5fd=hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd., 1588230740=hbase:meta,,1.1588230740, 02d6639888ca39d8ac3cb793f09913bc=TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc.} 2023-05-22 03:56:12,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:56:12,727 INFO [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(3303): Received CLOSE for 47b192f2be256dfdce2a5a9b181dd5fd 2023-05-22 03:56:12,727 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-05-22 03:56:12,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:56:12,727 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-05-22 03:56:12,727 DEBUG [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(1504): Waiting on 02d6639888ca39d8ac3cb793f09913bc, 1588230740, 47b192f2be256dfdce2a5a9b181dd5fd 2023-05-22 03:56:12,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. after waiting 0 ms 2023-05-22 03:56:12,727 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-05-22 03:56:12,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:56:12,728 DEBUG [regionserver/jenkins-hbase20:0.logRoller] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-05-22 03:56:12,728 DEBUG [regionserver/jenkins-hbase20:0.logRoller] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-05-22 03:56:12,728 DEBUG [regionserver/jenkins-hbase20:0.logRoller] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-05-22 03:56:12,728 INFO [regionserver/jenkins-hbase20:0.logRoller] regionserver.HRegionServer(2555): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1102053376, "init": 524288000, "max": 2051014656, "used": 324197128 }, "NonHeapMemoryUsage": { "committed": 139026432, "init": 2555904, "max": -1, "used": 136531192 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [], "beans": [], "beans": [] } 2023-05-22 03:56:12,728 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-05-22 03:56:12,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 02d6639888ca39d8ac3cb793f09913bc: 2023-05-22 03:56:12,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2539): Abort already in progress. Ignoring the current request with reason: Unrecoverable exception while closing TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:56:12,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 47b192f2be256dfdce2a5a9b181dd5fd, disabling compactions & flushes 2023-05-22 03:56:12,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:56:12,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44087] master.MasterRpcServices(609): jenkins-hbase20.apache.org,40121,1684727718365 reported a fatal error: ***** ABORTING region server jenkins-hbase20.apache.org,40121,1684727718365: Failed log close in log roller ***** Cause: org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException: hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/WALs/jenkins-hbase20.apache.org,40121,1684727718365/jenkins-hbase20.apache.org%2C40121%2C1684727718365.1684727760628, unflushedEntries=0 at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doReplaceWriter(FSHLog.java:427) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doReplaceWriter(FSHLog.java:70) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.replaceWriter(AbstractFSWAL.java:828) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:884) at org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:304) at org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:211) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unexpected BlockUCState: BP-69416852-148.251.75.209-1684727717821:blk_1073741841_1021 is UNDER_RECOVERY but not UNDER_CONSTRUCTION at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:4886) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:4955) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:954) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:992) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:507) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1034) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2833) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1540) at org.apache.hadoop.ipc.Client.call(Client.java:1486) at org.apache.hadoop.ipc.Client.call(Client.java:1385) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy30.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:918) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy33.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:361) at com.sun.proxy.$Proxy34.updateBlockForPipeline(Unknown Source) at org.apache.hadoop.hdfs.DataStreamer.updateBlockForPipeline(DataStreamer.java:1602) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1479) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1244) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:663) 2023-05-22 03:56:12,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:56:12,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. after waiting 0 ms 2023-05-22 03:56:12,729 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 3028 in region hbase:meta,,1.1588230740 2023-05-22 03:56:12,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:56:12,729 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-05-22 03:56:12,729 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-05-22 03:56:12,730 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-05-22 03:56:12,730 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-05-22 03:56:12,730 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(197): WAL FSHLog jenkins-hbase20.apache.org%2C40121%2C1684727718365.meta:.meta(num 1684727718938) roll requested 2023-05-22 03:56:12,730 DEBUG [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractFSWAL(874): WAL closed. Skipping rolling of writer 2023-05-22 03:56:12,730 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:56:12,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:56:12,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 47b192f2be256dfdce2a5a9b181dd5fd: 2023-05-22 03:56:12,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1684727718998.47b192f2be256dfdce2a5a9b181dd5fd. 2023-05-22 03:56:12,928 INFO [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(3303): Received CLOSE for 02d6639888ca39d8ac3cb793f09913bc 2023-05-22 03:56:12,929 DEBUG [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(1504): Waiting on 02d6639888ca39d8ac3cb793f09913bc 2023-05-22 03:56:12,929 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 02d6639888ca39d8ac3cb793f09913bc, disabling compactions & flushes 2023-05-22 03:56:12,929 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:56:12,929 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:56:12,929 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. after waiting 0 ms 2023-05-22 03:56:12,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:56:12,931 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 4304 in region TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:56:12,932 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:56:12,932 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 02d6639888ca39d8ac3cb793f09913bc: 2023-05-22 03:56:12,932 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed TestLogRolling-testLogRollOnPipelineRestart,,1684727719513.02d6639888ca39d8ac3cb793f09913bc. 2023-05-22 03:56:13,129 INFO [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,40121,1684727718365; all regions closed. 2023-05-22 03:56:13,129 DEBUG [RS:0;jenkins-hbase20:40121] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:56:13,130 INFO [RS:0;jenkins-hbase20:40121] regionserver.LeaseManager(133): Closed leases 2023-05-22 03:56:13,130 INFO [RS:0;jenkins-hbase20:40121] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-05-22 03:56:13,131 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-05-22 03:56:13,133 INFO [RS:0;jenkins-hbase20:40121] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:40121 2023-05-22 03:56:13,139 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): regionserver:40121-0x101714ef9d00001, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,40121,1684727718365 2023-05-22 03:56:13,139 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:56:13,139 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): regionserver:40121-0x101714ef9d00001, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:56:13,140 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,40121,1684727718365] 2023-05-22 03:56:13,140 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,40121,1684727718365; numProcessing=1 2023-05-22 03:56:13,141 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,40121,1684727718365 already deleted, retry=false 2023-05-22 03:56:13,141 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,40121,1684727718365 expired; onlineServers=0 2023-05-22 03:56:13,141 INFO [RegionServerTracker-0] regionserver.HRegionServer(2295): ***** STOPPING region server 'jenkins-hbase20.apache.org,44087,1684727718327' ***** 2023-05-22 03:56:13,141 INFO [RegionServerTracker-0] regionserver.HRegionServer(2309): STOPPED: Cluster shutdown set; onlineServer=0 2023-05-22 03:56:13,141 DEBUG [M:0;jenkins-hbase20:44087] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3346fae1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-05-22 03:56:13,141 INFO [M:0;jenkins-hbase20:44087] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,44087,1684727718327 2023-05-22 03:56:13,141 INFO [M:0;jenkins-hbase20:44087] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,44087,1684727718327; all regions closed. 2023-05-22 03:56:13,141 DEBUG [M:0;jenkins-hbase20:44087] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:56:13,142 DEBUG [M:0;jenkins-hbase20:44087] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-05-22 03:56:13,142 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-05-22 03:56:13,142 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727718562] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727718562,5,FailOnTimeoutGroup] 2023-05-22 03:56:13,143 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727718562] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727718562,5,FailOnTimeoutGroup] 2023-05-22 03:56:13,142 DEBUG [M:0;jenkins-hbase20:44087] cleaner.HFileCleaner(317): Stopping file delete threads 2023-05-22 03:56:13,144 INFO [M:0;jenkins-hbase20:44087] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-05-22 03:56:13,144 INFO [M:0;jenkins-hbase20:44087] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-05-22 03:56:13,144 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-05-22 03:56:13,144 INFO [M:0;jenkins-hbase20:44087] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-05-22 03:56:13,144 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:56:13,145 DEBUG [M:0;jenkins-hbase20:44087] master.HMaster(1512): Stopping service threads 2023-05-22 03:56:13,145 INFO [M:0;jenkins-hbase20:44087] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-05-22 03:56:13,145 ERROR [M:0;jenkins-hbase20:44087] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-05-22 03:56:13,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:56:13,145 INFO [M:0;jenkins-hbase20:44087] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-05-22 03:56:13,146 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-05-22 03:56:13,146 DEBUG [M:0;jenkins-hbase20:44087] zookeeper.ZKUtil(398): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-05-22 03:56:13,146 WARN [M:0;jenkins-hbase20:44087] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-05-22 03:56:13,146 INFO [M:0;jenkins-hbase20:44087] assignment.AssignmentManager(315): Stopping assignment manager 2023-05-22 03:56:13,147 INFO [M:0;jenkins-hbase20:44087] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-05-22 03:56:13,147 DEBUG [M:0;jenkins-hbase20:44087] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-05-22 03:56:13,147 INFO [M:0;jenkins-hbase20:44087] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:56:13,147 DEBUG [M:0;jenkins-hbase20:44087] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:56:13,148 DEBUG [M:0;jenkins-hbase20:44087] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-05-22 03:56:13,148 DEBUG [M:0;jenkins-hbase20:44087] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:56:13,148 INFO [M:0;jenkins-hbase20:44087] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=38.20 KB heapSize=45.83 KB 2023-05-22 03:56:13,169 INFO [M:0;jenkins-hbase20:44087] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.20 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/57d3976b4176403ba67d8b6cdea78617 2023-05-22 03:56:13,177 DEBUG [M:0;jenkins-hbase20:44087] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/57d3976b4176403ba67d8b6cdea78617 as hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/57d3976b4176403ba67d8b6cdea78617 2023-05-22 03:56:13,185 INFO [M:0;jenkins-hbase20:44087] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37871/user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/57d3976b4176403ba67d8b6cdea78617, entries=11, sequenceid=92, filesize=7.0 K 2023-05-22 03:56:13,186 INFO [M:0;jenkins-hbase20:44087] regionserver.HRegion(2948): Finished flush of dataSize ~38.20 KB/39113, heapSize ~45.81 KB/46912, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 38ms, sequenceid=92, compaction requested=false 2023-05-22 03:56:13,187 INFO [M:0;jenkins-hbase20:44087] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:56:13,188 DEBUG [M:0;jenkins-hbase20:44087] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:56:13,188 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/67cbbb8c-c30e-193b-80fd-d3934a6a603d/MasterData/WALs/jenkins-hbase20.apache.org,44087,1684727718327 2023-05-22 03:56:13,191 INFO [M:0;jenkins-hbase20:44087] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-05-22 03:56:13,191 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-05-22 03:56:13,191 INFO [M:0;jenkins-hbase20:44087] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:44087 2023-05-22 03:56:13,193 DEBUG [M:0;jenkins-hbase20:44087] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,44087,1684727718327 already deleted, retry=false 2023-05-22 03:56:13,240 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): regionserver:40121-0x101714ef9d00001, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:56:13,240 INFO [RS:0;jenkins-hbase20:40121] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,40121,1684727718365; zookeeper connection closed. 2023-05-22 03:56:13,240 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): regionserver:40121-0x101714ef9d00001, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:56:13,241 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@46966bf5] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@46966bf5 2023-05-22 03:56:13,247 INFO [Listener at localhost.localdomain/39159] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-05-22 03:56:13,340 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:56:13,341 INFO [M:0;jenkins-hbase20:44087] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,44087,1684727718327; zookeeper connection closed. 2023-05-22 03:56:13,341 DEBUG [Listener at localhost.localdomain/35103-EventThread] zookeeper.ZKWatcher(600): master:44087-0x101714ef9d00000, quorum=127.0.0.1:49349, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:56:13,343 WARN [Listener at localhost.localdomain/39159] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:56:13,352 INFO [Listener at localhost.localdomain/39159] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:56:13,462 WARN [BP-69416852-148.251.75.209-1684727717821 heartbeating to localhost.localdomain/127.0.0.1:37871] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:56:13,462 WARN [BP-69416852-148.251.75.209-1684727717821 heartbeating to localhost.localdomain/127.0.0.1:37871] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-69416852-148.251.75.209-1684727717821 (Datanode Uuid 5181a363-6d97-47ef-86f3-4730977e7fb9) service to localhost.localdomain/127.0.0.1:37871 2023-05-22 03:56:13,463 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data3/current/BP-69416852-148.251.75.209-1684727717821] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:56:13,464 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data4/current/BP-69416852-148.251.75.209-1684727717821] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:56:13,466 WARN [Listener at localhost.localdomain/39159] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:56:13,470 INFO [Listener at localhost.localdomain/39159] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:56:13,577 WARN [BP-69416852-148.251.75.209-1684727717821 heartbeating to localhost.localdomain/127.0.0.1:37871] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:56:13,577 WARN [BP-69416852-148.251.75.209-1684727717821 heartbeating to localhost.localdomain/127.0.0.1:37871] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-69416852-148.251.75.209-1684727717821 (Datanode Uuid 6eb1cf74-c837-4445-a555-7e1f0435d520) service to localhost.localdomain/127.0.0.1:37871 2023-05-22 03:56:13,579 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data1/current/BP-69416852-148.251.75.209-1684727717821] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:56:13,580 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/cluster_917d31fd-c192-8ca1-8c71-9a06472d1c35/dfs/data/data2/current/BP-69416852-148.251.75.209-1684727717821] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:56:13,594 INFO [Listener at localhost.localdomain/39159] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-05-22 03:56:13,704 INFO [Listener at localhost.localdomain/39159] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-05-22 03:56:13,716 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-05-22 03:56:13,725 INFO [Listener at localhost.localdomain/39159] hbase.ResourceChecker(175): after: regionserver.wal.TestLogRolling#testLogRollOnPipelineRestart Thread=88 (was 78) Potentially hanging thread: nioEventLoopGroup-28-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-27-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-26-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost.localdomain/39159 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (551965115) connection to localhost.localdomain/127.0.0.1:37871 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: LeaseRenewer:jenkins@localhost.localdomain:37871 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.3@localhost.localdomain:37871 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-29-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (551965115) connection to localhost.localdomain/127.0.0.1:37871 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-28-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-27-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-26-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-26-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-28-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (551965115) connection to localhost.localdomain/127.0.0.1:37871 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-29-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-29-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-27-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=461 (was 472), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=110 (was 154), ProcessCount=175 (was 168) - ProcessCount LEAK? -, AvailableMemoryMB=6909 (was 7386) 2023-05-22 03:56:13,732 INFO [Listener at localhost.localdomain/39159] hbase.ResourceChecker(147): before: regionserver.wal.TestLogRolling#testCompactionRecordDoesntBlockRolling Thread=88, OpenFileDescriptor=461, MaxFileDescriptor=60000, SystemLoadAverage=110, ProcessCount=175, AvailableMemoryMB=6909 2023-05-22 03:56:13,732 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-05-22 03:56:13,733 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/hadoop.log.dir so I do NOT create it in target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167 2023-05-22 03:56:13,733 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4119f273-4961-f2bd-653b-e993b95a99fc/hadoop.tmp.dir so I do NOT create it in target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167 2023-05-22 03:56:13,733 INFO [Listener at localhost.localdomain/39159] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/cluster_d6f4cf05-5962-2ada-e7eb-e39cba3bab08, deleteOnExit=true 2023-05-22 03:56:13,733 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-05-22 03:56:13,733 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/test.cache.data in system properties and HBase conf 2023-05-22 03:56:13,733 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/hadoop.tmp.dir in system properties and HBase conf 2023-05-22 03:56:13,733 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/hadoop.log.dir in system properties and HBase conf 2023-05-22 03:56:13,734 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/mapreduce.cluster.local.dir in system properties and HBase conf 2023-05-22 03:56:13,734 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-05-22 03:56:13,734 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-05-22 03:56:13,734 DEBUG [Listener at localhost.localdomain/39159] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-05-22 03:56:13,734 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-05-22 03:56:13,734 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-05-22 03:56:13,734 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-05-22 03:56:13,734 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-05-22 03:56:13,735 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-05-22 03:56:13,735 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-05-22 03:56:13,735 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-05-22 03:56:13,735 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/dfs.journalnode.edits.dir in system properties and HBase conf 2023-05-22 03:56:13,735 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-05-22 03:56:13,735 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/nfs.dump.dir in system properties and HBase conf 2023-05-22 03:56:13,735 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/java.io.tmpdir in system properties and HBase conf 2023-05-22 03:56:13,735 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/dfs.journalnode.edits.dir in system properties and HBase conf 2023-05-22 03:56:13,735 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-05-22 03:56:13,736 INFO [Listener at localhost.localdomain/39159] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-05-22 03:56:13,737 WARN [Listener at localhost.localdomain/39159] blockmanagement.DatanodeManager(362): The given interval for marking stale datanode = 30000, which is larger than heartbeat expire interval 20000. 2023-05-22 03:56:13,738 WARN [Listener at localhost.localdomain/39159] conf.Configuration(1701): No unit for dfs.heartbeat.interval(1) assuming SECONDS 2023-05-22 03:56:13,738 WARN [Listener at localhost.localdomain/39159] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-05-22 03:56:13,760 WARN [Listener at localhost.localdomain/39159] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:56:13,762 INFO [Listener at localhost.localdomain/39159] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:56:13,767 INFO [Listener at localhost.localdomain/39159] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/java.io.tmpdir/Jetty_localhost_localdomain_46761_hdfs____emygy2/webapp 2023-05-22 03:56:13,838 INFO [Listener at localhost.localdomain/39159] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:46761 2023-05-22 03:56:13,839 WARN [Listener at localhost.localdomain/39159] blockmanagement.DatanodeManager(362): The given interval for marking stale datanode = 30000, which is larger than heartbeat expire interval 20000. 2023-05-22 03:56:13,840 WARN [Listener at localhost.localdomain/39159] conf.Configuration(1701): No unit for dfs.heartbeat.interval(1) assuming SECONDS 2023-05-22 03:56:13,840 WARN [Listener at localhost.localdomain/39159] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-05-22 03:56:13,864 WARN [Listener at localhost.localdomain/42989] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:56:13,871 WARN [Listener at localhost.localdomain/42989] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:56:13,874 WARN [Listener at localhost.localdomain/42989] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:56:13,875 INFO [Listener at localhost.localdomain/42989] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:56:13,880 INFO [Listener at localhost.localdomain/42989] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/java.io.tmpdir/Jetty_localhost_44165_datanode____wgis7/webapp 2023-05-22 03:56:13,953 INFO [Listener at localhost.localdomain/42989] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44165 2023-05-22 03:56:13,958 WARN [Listener at localhost.localdomain/36403] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:56:13,966 WARN [Listener at localhost.localdomain/36403] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:56:13,969 WARN [Listener at localhost.localdomain/36403] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:56:13,970 INFO [Listener at localhost.localdomain/36403] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:56:13,973 INFO [Listener at localhost.localdomain/36403] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/java.io.tmpdir/Jetty_localhost_43131_datanode____.h6yt2h/webapp 2023-05-22 03:56:14,013 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd9530525eae1df48: Processing first storage report for DS-2e428832-7c1d-4bb3-a716-f1edebf826ea from datanode 146c2160-e757-4007-9eaa-d82aaee88864 2023-05-22 03:56:14,013 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd9530525eae1df48: from storage DS-2e428832-7c1d-4bb3-a716-f1edebf826ea node DatanodeRegistration(127.0.0.1:42319, datanodeUuid=146c2160-e757-4007-9eaa-d82aaee88864, infoPort=44521, infoSecurePort=0, ipcPort=36403, storageInfo=lv=-57;cid=testClusterID;nsid=439344065;c=1684727773740), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:56:14,014 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd9530525eae1df48: Processing first storage report for DS-4476fc78-c6d4-4e9c-8c08-7fb45fe218b7 from datanode 146c2160-e757-4007-9eaa-d82aaee88864 2023-05-22 03:56:14,014 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd9530525eae1df48: from storage DS-4476fc78-c6d4-4e9c-8c08-7fb45fe218b7 node DatanodeRegistration(127.0.0.1:42319, datanodeUuid=146c2160-e757-4007-9eaa-d82aaee88864, infoPort=44521, infoSecurePort=0, ipcPort=36403, storageInfo=lv=-57;cid=testClusterID;nsid=439344065;c=1684727773740), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:56:14,053 INFO [Listener at localhost.localdomain/36403] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43131 2023-05-22 03:56:14,059 WARN [Listener at localhost.localdomain/35753] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:56:14,107 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa1039be6474efc16: Processing first storage report for DS-9e3e0a3e-2325-44be-89a8-4152a9f276f7 from datanode 41a3d320-5358-4527-be98-ce2d395bea19 2023-05-22 03:56:14,108 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa1039be6474efc16: from storage DS-9e3e0a3e-2325-44be-89a8-4152a9f276f7 node DatanodeRegistration(127.0.0.1:46309, datanodeUuid=41a3d320-5358-4527-be98-ce2d395bea19, infoPort=35591, infoSecurePort=0, ipcPort=35753, storageInfo=lv=-57;cid=testClusterID;nsid=439344065;c=1684727773740), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:56:14,108 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa1039be6474efc16: Processing first storage report for DS-64791ad8-17e6-433f-9977-9497064dca1d from datanode 41a3d320-5358-4527-be98-ce2d395bea19 2023-05-22 03:56:14,108 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa1039be6474efc16: from storage DS-64791ad8-17e6-433f-9977-9497064dca1d node DatanodeRegistration(127.0.0.1:46309, datanodeUuid=41a3d320-5358-4527-be98-ce2d395bea19, infoPort=35591, infoSecurePort=0, ipcPort=35753, storageInfo=lv=-57;cid=testClusterID;nsid=439344065;c=1684727773740), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:56:14,167 DEBUG [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167 2023-05-22 03:56:14,172 INFO [Listener at localhost.localdomain/35753] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/cluster_d6f4cf05-5962-2ada-e7eb-e39cba3bab08/zookeeper_0, clientPort=58947, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/cluster_d6f4cf05-5962-2ada-e7eb-e39cba3bab08/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/cluster_d6f4cf05-5962-2ada-e7eb-e39cba3bab08/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-05-22 03:56:14,174 INFO [Listener at localhost.localdomain/35753] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58947 2023-05-22 03:56:14,174 INFO [Listener at localhost.localdomain/35753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:56:14,175 INFO [Listener at localhost.localdomain/35753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:56:14,187 INFO [Listener at localhost.localdomain/35753] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e with version=8 2023-05-22 03:56:14,187 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/hbase-staging 2023-05-22 03:56:14,189 INFO [Listener at localhost.localdomain/35753] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-05-22 03:56:14,189 INFO [Listener at localhost.localdomain/35753] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:56:14,189 INFO [Listener at localhost.localdomain/35753] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-05-22 03:56:14,189 INFO [Listener at localhost.localdomain/35753] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-05-22 03:56:14,189 INFO [Listener at localhost.localdomain/35753] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:56:14,189 INFO [Listener at localhost.localdomain/35753] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-05-22 03:56:14,189 INFO [Listener at localhost.localdomain/35753] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-05-22 03:56:14,190 INFO [Listener at localhost.localdomain/35753] ipc.NettyRpcServer(120): Bind to /148.251.75.209:39195 2023-05-22 03:56:14,191 INFO [Listener at localhost.localdomain/35753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:56:14,192 INFO [Listener at localhost.localdomain/35753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:56:14,192 INFO [Listener at localhost.localdomain/35753] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39195 connecting to ZooKeeper ensemble=127.0.0.1:58947 2023-05-22 03:56:14,197 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:391950x0, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-05-22 03:56:14,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:39195-0x101714fd4060000 connected 2023-05-22 03:56:14,209 DEBUG [Listener at localhost.localdomain/35753] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:56:14,209 DEBUG [Listener at localhost.localdomain/35753] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:56:14,210 DEBUG [Listener at localhost.localdomain/35753] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-05-22 03:56:14,211 DEBUG [Listener at localhost.localdomain/35753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39195 2023-05-22 03:56:14,212 DEBUG [Listener at localhost.localdomain/35753] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39195 2023-05-22 03:56:14,212 DEBUG [Listener at localhost.localdomain/35753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39195 2023-05-22 03:56:14,212 DEBUG [Listener at localhost.localdomain/35753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39195 2023-05-22 03:56:14,212 DEBUG [Listener at localhost.localdomain/35753] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39195 2023-05-22 03:56:14,213 INFO [Listener at localhost.localdomain/35753] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e, hbase.cluster.distributed=false 2023-05-22 03:56:14,227 INFO [Listener at localhost.localdomain/35753] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-05-22 03:56:14,228 INFO [Listener at localhost.localdomain/35753] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:56:14,228 INFO [Listener at localhost.localdomain/35753] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-05-22 03:56:14,228 INFO [Listener at localhost.localdomain/35753] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-05-22 03:56:14,228 INFO [Listener at localhost.localdomain/35753] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:56:14,228 INFO [Listener at localhost.localdomain/35753] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-05-22 03:56:14,228 INFO [Listener at localhost.localdomain/35753] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-05-22 03:56:14,230 INFO [Listener at localhost.localdomain/35753] ipc.NettyRpcServer(120): Bind to /148.251.75.209:40647 2023-05-22 03:56:14,230 INFO [Listener at localhost.localdomain/35753] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-05-22 03:56:14,231 DEBUG [Listener at localhost.localdomain/35753] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-05-22 03:56:14,231 INFO [Listener at localhost.localdomain/35753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:56:14,232 INFO [Listener at localhost.localdomain/35753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:56:14,233 INFO [Listener at localhost.localdomain/35753] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40647 connecting to ZooKeeper ensemble=127.0.0.1:58947 2023-05-22 03:56:14,236 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:406470x0, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-05-22 03:56:14,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:40647-0x101714fd4060001 connected 2023-05-22 03:56:14,237 DEBUG [Listener at localhost.localdomain/35753] zookeeper.ZKUtil(164): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:56:14,238 DEBUG [Listener at localhost.localdomain/35753] zookeeper.ZKUtil(164): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:56:14,238 DEBUG [Listener at localhost.localdomain/35753] zookeeper.ZKUtil(164): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-05-22 03:56:14,239 DEBUG [Listener at localhost.localdomain/35753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40647 2023-05-22 03:56:14,239 DEBUG [Listener at localhost.localdomain/35753] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40647 2023-05-22 03:56:14,239 DEBUG [Listener at localhost.localdomain/35753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40647 2023-05-22 03:56:14,239 DEBUG [Listener at localhost.localdomain/35753] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40647 2023-05-22 03:56:14,240 DEBUG [Listener at localhost.localdomain/35753] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40647 2023-05-22 03:56:14,241 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,39195,1684727774188 2023-05-22 03:56:14,242 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-05-22 03:56:14,242 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,39195,1684727774188 2023-05-22 03:56:14,243 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-05-22 03:56:14,243 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-05-22 03:56:14,243 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:56:14,244 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-05-22 03:56:14,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-05-22 03:56:14,245 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,39195,1684727774188 from backup master directory 2023-05-22 03:56:14,246 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,39195,1684727774188 2023-05-22 03:56:14,246 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-05-22 03:56:14,246 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-05-22 03:56:14,246 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,39195,1684727774188 2023-05-22 03:56:14,259 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/hbase.id with ID: 7d788674-7798-480f-a803-5c3ee9aad7b9 2023-05-22 03:56:14,274 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:56:14,276 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:56:14,286 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0e8b6c66 to 127.0.0.1:58947 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:56:14,291 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25d843d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:56:14,292 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-05-22 03:56:14,292 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-05-22 03:56:14,293 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:56:14,294 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/MasterData/data/master/store-tmp 2023-05-22 03:56:14,304 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:56:14,304 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-05-22 03:56:14,304 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:56:14,304 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:56:14,304 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-05-22 03:56:14,305 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:56:14,305 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:56:14,305 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:56:14,305 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/MasterData/WALs/jenkins-hbase20.apache.org,39195,1684727774188 2023-05-22 03:56:14,308 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C39195%2C1684727774188, suffix=, logDir=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/MasterData/WALs/jenkins-hbase20.apache.org,39195,1684727774188, archiveDir=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/MasterData/oldWALs, maxLogs=10 2023-05-22 03:56:14,314 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/MasterData/WALs/jenkins-hbase20.apache.org,39195,1684727774188/jenkins-hbase20.apache.org%2C39195%2C1684727774188.1684727774309 2023-05-22 03:56:14,314 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42319,DS-2e428832-7c1d-4bb3-a716-f1edebf826ea,DISK], DatanodeInfoWithStorage[127.0.0.1:46309,DS-9e3e0a3e-2325-44be-89a8-4152a9f276f7,DISK]] 2023-05-22 03:56:14,314 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:56:14,315 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:56:14,315 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:56:14,315 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:56:14,318 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:56:14,319 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-05-22 03:56:14,320 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-05-22 03:56:14,320 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:56:14,321 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:56:14,321 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:56:14,324 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:56:14,326 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:56:14,326 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=722638, jitterRate=-0.08111894130706787}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:56:14,326 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:56:14,327 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-05-22 03:56:14,328 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-05-22 03:56:14,328 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-05-22 03:56:14,328 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-05-22 03:56:14,328 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-05-22 03:56:14,329 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-05-22 03:56:14,329 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-05-22 03:56:14,329 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-05-22 03:56:14,330 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-05-22 03:56:14,342 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-05-22 03:56:14,342 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-22 03:56:14,342 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-05-22 03:56:14,342 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-05-22 03:56:14,343 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-05-22 03:56:14,345 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:56:14,345 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-05-22 03:56:14,345 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-05-22 03:56:14,346 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-05-22 03:56:14,347 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-05-22 03:56:14,347 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:56:14,347 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-05-22 03:56:14,347 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,39195,1684727774188, sessionid=0x101714fd4060000, setting cluster-up flag (Was=false) 2023-05-22 03:56:14,350 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:56:14,352 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-05-22 03:56:14,353 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,39195,1684727774188 2023-05-22 03:56:14,354 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:56:14,356 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-05-22 03:56:14,357 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,39195,1684727774188 2023-05-22 03:56:14,357 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/.hbase-snapshot/.tmp 2023-05-22 03:56:14,360 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-05-22 03:56:14,360 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:56:14,361 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:56:14,361 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:56:14,361 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:56:14,361 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-05-22 03:56:14,361 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:56:14,361 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-05-22 03:56:14,361 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:56:14,363 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1684727804362 2023-05-22 03:56:14,363 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-05-22 03:56:14,363 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-05-22 03:56:14,363 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-05-22 03:56:14,363 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-05-22 03:56:14,363 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-05-22 03:56:14,363 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-05-22 03:56:14,363 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-05-22 03:56:14,364 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-05-22 03:56:14,364 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-05-22 03:56:14,364 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-05-22 03:56:14,364 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-05-22 03:56:14,365 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-05-22 03:56:14,365 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-05-22 03:56:14,365 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-05-22 03:56:14,365 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727774365,5,FailOnTimeoutGroup] 2023-05-22 03:56:14,365 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727774365,5,FailOnTimeoutGroup] 2023-05-22 03:56:14,365 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-05-22 03:56:14,365 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-05-22 03:56:14,365 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-05-22 03:56:14,366 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-05-22 03:56:14,366 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-05-22 03:56:14,377 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-05-22 03:56:14,378 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-05-22 03:56:14,378 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e 2023-05-22 03:56:14,387 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:56:14,388 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-05-22 03:56:14,389 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/info 2023-05-22 03:56:14,390 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-05-22 03:56:14,391 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:56:14,391 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-05-22 03:56:14,392 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:56:14,392 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-05-22 03:56:14,393 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:56:14,393 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-05-22 03:56:14,394 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/table 2023-05-22 03:56:14,394 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-05-22 03:56:14,395 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:56:14,396 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740 2023-05-22 03:56:14,396 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740 2023-05-22 03:56:14,399 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-05-22 03:56:14,400 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-05-22 03:56:14,403 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:56:14,403 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=855145, jitterRate=0.08737346529960632}}}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-05-22 03:56:14,403 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-05-22 03:56:14,403 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-05-22 03:56:14,403 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-05-22 03:56:14,403 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-05-22 03:56:14,403 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-05-22 03:56:14,403 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-05-22 03:56:14,404 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-05-22 03:56:14,404 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-05-22 03:56:14,405 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-05-22 03:56:14,405 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-05-22 03:56:14,405 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-05-22 03:56:14,407 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-05-22 03:56:14,408 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-05-22 03:56:14,442 INFO [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(951): ClusterId : 7d788674-7798-480f-a803-5c3ee9aad7b9 2023-05-22 03:56:14,444 DEBUG [RS:0;jenkins-hbase20:40647] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-05-22 03:56:14,446 DEBUG [RS:0;jenkins-hbase20:40647] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-05-22 03:56:14,446 DEBUG [RS:0;jenkins-hbase20:40647] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-05-22 03:56:14,449 DEBUG [RS:0;jenkins-hbase20:40647] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-05-22 03:56:14,450 DEBUG [RS:0;jenkins-hbase20:40647] zookeeper.ReadOnlyZKClient(139): Connect 0x170edb77 to 127.0.0.1:58947 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:56:14,462 DEBUG [RS:0;jenkins-hbase20:40647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19c937e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:56:14,462 DEBUG [RS:0;jenkins-hbase20:40647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1df6b007, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-05-22 03:56:14,473 DEBUG [RS:0;jenkins-hbase20:40647] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:40647 2023-05-22 03:56:14,473 INFO [RS:0;jenkins-hbase20:40647] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-05-22 03:56:14,473 INFO [RS:0;jenkins-hbase20:40647] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-05-22 03:56:14,473 DEBUG [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(1022): About to register with Master. 2023-05-22 03:56:14,474 INFO [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(2809): reportForDuty to master=jenkins-hbase20.apache.org,39195,1684727774188 with isa=jenkins-hbase20.apache.org/148.251.75.209:40647, startcode=1684727774227 2023-05-22 03:56:14,474 DEBUG [RS:0;jenkins-hbase20:40647] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-05-22 03:56:14,480 INFO [RS-EventLoopGroup-10-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40267, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-05-22 03:56:14,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:14,481 DEBUG [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e 2023-05-22 03:56:14,481 DEBUG [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:42989 2023-05-22 03:56:14,481 DEBUG [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-05-22 03:56:14,483 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:56:14,483 DEBUG [RS:0;jenkins-hbase20:40647] zookeeper.ZKUtil(162): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:14,484 WARN [RS:0;jenkins-hbase20:40647] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-05-22 03:56:14,484 INFO [RS:0;jenkins-hbase20:40647] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:56:14,484 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,40647,1684727774227] 2023-05-22 03:56:14,484 DEBUG [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(1946): logDir=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:14,488 DEBUG [RS:0;jenkins-hbase20:40647] zookeeper.ZKUtil(162): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:14,489 DEBUG [RS:0;jenkins-hbase20:40647] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-05-22 03:56:14,489 INFO [RS:0;jenkins-hbase20:40647] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-05-22 03:56:14,490 INFO [RS:0;jenkins-hbase20:40647] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-05-22 03:56:14,490 INFO [RS:0;jenkins-hbase20:40647] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-05-22 03:56:14,490 INFO [RS:0;jenkins-hbase20:40647] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:56:14,490 INFO [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-05-22 03:56:14,492 INFO [RS:0;jenkins-hbase20:40647] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-05-22 03:56:14,492 DEBUG [RS:0;jenkins-hbase20:40647] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:56:14,492 DEBUG [RS:0;jenkins-hbase20:40647] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:56:14,492 DEBUG [RS:0;jenkins-hbase20:40647] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:56:14,492 DEBUG [RS:0;jenkins-hbase20:40647] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:56:14,492 DEBUG [RS:0;jenkins-hbase20:40647] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:56:14,492 DEBUG [RS:0;jenkins-hbase20:40647] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-05-22 03:56:14,493 DEBUG [RS:0;jenkins-hbase20:40647] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:56:14,493 DEBUG [RS:0;jenkins-hbase20:40647] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:56:14,493 DEBUG [RS:0;jenkins-hbase20:40647] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:56:14,493 DEBUG [RS:0;jenkins-hbase20:40647] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:56:14,494 INFO [RS:0;jenkins-hbase20:40647] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-05-22 03:56:14,494 INFO [RS:0;jenkins-hbase20:40647] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-05-22 03:56:14,494 INFO [RS:0;jenkins-hbase20:40647] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-05-22 03:56:14,506 INFO [RS:0;jenkins-hbase20:40647] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-05-22 03:56:14,506 INFO [RS:0;jenkins-hbase20:40647] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40647,1684727774227-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:56:14,517 INFO [RS:0;jenkins-hbase20:40647] regionserver.Replication(203): jenkins-hbase20.apache.org,40647,1684727774227 started 2023-05-22 03:56:14,517 INFO [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,40647,1684727774227, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:40647, sessionid=0x101714fd4060001 2023-05-22 03:56:14,517 DEBUG [RS:0;jenkins-hbase20:40647] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-05-22 03:56:14,517 DEBUG [RS:0;jenkins-hbase20:40647] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:14,517 DEBUG [RS:0;jenkins-hbase20:40647] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,40647,1684727774227' 2023-05-22 03:56:14,517 DEBUG [RS:0;jenkins-hbase20:40647] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:56:14,517 DEBUG [RS:0;jenkins-hbase20:40647] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:56:14,518 DEBUG [RS:0;jenkins-hbase20:40647] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-05-22 03:56:14,518 DEBUG [RS:0;jenkins-hbase20:40647] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-05-22 03:56:14,518 DEBUG [RS:0;jenkins-hbase20:40647] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:14,518 DEBUG [RS:0;jenkins-hbase20:40647] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,40647,1684727774227' 2023-05-22 03:56:14,518 DEBUG [RS:0;jenkins-hbase20:40647] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-05-22 03:56:14,518 DEBUG [RS:0;jenkins-hbase20:40647] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-05-22 03:56:14,519 DEBUG [RS:0;jenkins-hbase20:40647] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-05-22 03:56:14,519 INFO [RS:0;jenkins-hbase20:40647] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-05-22 03:56:14,519 INFO [RS:0;jenkins-hbase20:40647] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-05-22 03:56:14,558 DEBUG [jenkins-hbase20:39195] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-05-22 03:56:14,559 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,40647,1684727774227, state=OPENING 2023-05-22 03:56:14,561 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-05-22 03:56:14,562 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:56:14,563 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,40647,1684727774227}] 2023-05-22 03:56:14,563 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-05-22 03:56:14,621 INFO [RS:0;jenkins-hbase20:40647] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C40647%2C1684727774227, suffix=, logDir=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227, archiveDir=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/oldWALs, maxLogs=32 2023-05-22 03:56:14,633 INFO [RS:0;jenkins-hbase20:40647] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227/jenkins-hbase20.apache.org%2C40647%2C1684727774227.1684727774622 2023-05-22 03:56:14,633 DEBUG [RS:0;jenkins-hbase20:40647] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46309,DS-9e3e0a3e-2325-44be-89a8-4152a9f276f7,DISK], DatanodeInfoWithStorage[127.0.0.1:42319,DS-2e428832-7c1d-4bb3-a716-f1edebf826ea,DISK]] 2023-05-22 03:56:14,633 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-05-22 03:56:14,720 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:14,721 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-05-22 03:56:14,726 INFO [RS-EventLoopGroup-11-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50076, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-05-22 03:56:14,733 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-05-22 03:56:14,733 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:56:14,735 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C40647%2C1684727774227.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227, archiveDir=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/oldWALs, maxLogs=32 2023-05-22 03:56:14,743 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227/jenkins-hbase20.apache.org%2C40647%2C1684727774227.meta.1684727774735.meta 2023-05-22 03:56:14,743 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42319,DS-2e428832-7c1d-4bb3-a716-f1edebf826ea,DISK], DatanodeInfoWithStorage[127.0.0.1:46309,DS-9e3e0a3e-2325-44be-89a8-4152a9f276f7,DISK]] 2023-05-22 03:56:14,744 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:56:14,744 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-05-22 03:56:14,744 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-05-22 03:56:14,744 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-05-22 03:56:14,745 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-05-22 03:56:14,745 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:56:14,745 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-05-22 03:56:14,745 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-05-22 03:56:14,747 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-05-22 03:56:14,748 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/info 2023-05-22 03:56:14,748 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/info 2023-05-22 03:56:14,748 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-05-22 03:56:14,749 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:56:14,749 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-05-22 03:56:14,750 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:56:14,750 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:56:14,750 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-05-22 03:56:14,751 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:56:14,751 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-05-22 03:56:14,752 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/table 2023-05-22 03:56:14,752 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/table 2023-05-22 03:56:14,752 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-05-22 03:56:14,753 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:56:14,754 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740 2023-05-22 03:56:14,755 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740 2023-05-22 03:56:14,757 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-05-22 03:56:14,759 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-05-22 03:56:14,759 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=773049, jitterRate=-0.01701807975769043}}}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-05-22 03:56:14,760 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-05-22 03:56:14,763 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1684727774720 2023-05-22 03:56:14,768 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for hbase:meta,,1.1588230740 2023-05-22 03:56:14,769 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-05-22 03:56:14,769 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,40647,1684727774227, state=OPEN 2023-05-22 03:56:14,771 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-05-22 03:56:14,771 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-05-22 03:56:14,774 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-05-22 03:56:14,775 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,40647,1684727774227 in 208 msec 2023-05-22 03:56:14,778 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-05-22 03:56:14,778 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 370 msec 2023-05-22 03:56:14,781 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 421 msec 2023-05-22 03:56:14,781 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1684727774781, completionTime=-1 2023-05-22 03:56:14,781 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-05-22 03:56:14,781 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-05-22 03:56:14,785 DEBUG [hconnection-0x552bca40-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-05-22 03:56:14,787 INFO [RS-EventLoopGroup-11-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50080, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-05-22 03:56:14,788 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-05-22 03:56:14,789 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1684727834788 2023-05-22 03:56:14,789 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1684727894789 2023-05-22 03:56:14,789 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 7 msec 2023-05-22 03:56:14,795 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,39195,1684727774188-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:56:14,795 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,39195,1684727774188-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:56:14,795 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,39195,1684727774188-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:56:14,795 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:39195, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:56:14,795 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-05-22 03:56:14,795 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-05-22 03:56:14,795 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-05-22 03:56:14,796 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-05-22 03:56:14,797 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-05-22 03:56:14,798 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-05-22 03:56:14,799 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-05-22 03:56:14,802 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/.tmp/data/hbase/namespace/3b5b921e3e20a943a4962a5489af2f33 2023-05-22 03:56:14,803 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/.tmp/data/hbase/namespace/3b5b921e3e20a943a4962a5489af2f33 empty. 2023-05-22 03:56:14,804 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/.tmp/data/hbase/namespace/3b5b921e3e20a943a4962a5489af2f33 2023-05-22 03:56:14,804 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-05-22 03:56:14,820 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-05-22 03:56:14,821 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3b5b921e3e20a943a4962a5489af2f33, NAME => 'hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/.tmp 2023-05-22 03:56:14,832 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:56:14,832 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 3b5b921e3e20a943a4962a5489af2f33, disabling compactions & flushes 2023-05-22 03:56:14,832 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. 2023-05-22 03:56:14,832 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. 2023-05-22 03:56:14,832 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. after waiting 0 ms 2023-05-22 03:56:14,832 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. 2023-05-22 03:56:14,832 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. 2023-05-22 03:56:14,832 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 3b5b921e3e20a943a4962a5489af2f33: 2023-05-22 03:56:14,835 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-05-22 03:56:14,836 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727774836"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1684727774836"}]},"ts":"1684727774836"} 2023-05-22 03:56:14,839 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-05-22 03:56:14,840 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-05-22 03:56:14,840 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727774840"}]},"ts":"1684727774840"} 2023-05-22 03:56:14,842 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-05-22 03:56:14,846 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=3b5b921e3e20a943a4962a5489af2f33, ASSIGN}] 2023-05-22 03:56:14,849 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=3b5b921e3e20a943a4962a5489af2f33, ASSIGN 2023-05-22 03:56:14,850 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=3b5b921e3e20a943a4962a5489af2f33, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,40647,1684727774227; forceNewPlan=false, retain=false 2023-05-22 03:56:15,001 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=3b5b921e3e20a943a4962a5489af2f33, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:15,001 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727775001"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1684727775001"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1684727775001"}]},"ts":"1684727775001"} 2023-05-22 03:56:15,004 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 3b5b921e3e20a943a4962a5489af2f33, server=jenkins-hbase20.apache.org,40647,1684727774227}] 2023-05-22 03:56:15,161 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. 2023-05-22 03:56:15,161 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3b5b921e3e20a943a4962a5489af2f33, NAME => 'hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:56:15,161 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 3b5b921e3e20a943a4962a5489af2f33 2023-05-22 03:56:15,161 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:56:15,161 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 3b5b921e3e20a943a4962a5489af2f33 2023-05-22 03:56:15,161 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 3b5b921e3e20a943a4962a5489af2f33 2023-05-22 03:56:15,163 INFO [StoreOpener-3b5b921e3e20a943a4962a5489af2f33-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 3b5b921e3e20a943a4962a5489af2f33 2023-05-22 03:56:15,164 DEBUG [StoreOpener-3b5b921e3e20a943a4962a5489af2f33-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/namespace/3b5b921e3e20a943a4962a5489af2f33/info 2023-05-22 03:56:15,165 DEBUG [StoreOpener-3b5b921e3e20a943a4962a5489af2f33-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/namespace/3b5b921e3e20a943a4962a5489af2f33/info 2023-05-22 03:56:15,165 INFO [StoreOpener-3b5b921e3e20a943a4962a5489af2f33-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3b5b921e3e20a943a4962a5489af2f33 columnFamilyName info 2023-05-22 03:56:15,166 INFO [StoreOpener-3b5b921e3e20a943a4962a5489af2f33-1] regionserver.HStore(310): Store=3b5b921e3e20a943a4962a5489af2f33/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:56:15,166 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/namespace/3b5b921e3e20a943a4962a5489af2f33 2023-05-22 03:56:15,167 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/namespace/3b5b921e3e20a943a4962a5489af2f33 2023-05-22 03:56:15,169 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 3b5b921e3e20a943a4962a5489af2f33 2023-05-22 03:56:15,171 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/namespace/3b5b921e3e20a943a4962a5489af2f33/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:56:15,171 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 3b5b921e3e20a943a4962a5489af2f33; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=859887, jitterRate=0.09340342879295349}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:56:15,171 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 3b5b921e3e20a943a4962a5489af2f33: 2023-05-22 03:56:15,173 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33., pid=6, masterSystemTime=1684727775157 2023-05-22 03:56:15,176 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. 2023-05-22 03:56:15,176 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. 2023-05-22 03:56:15,176 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=3b5b921e3e20a943a4962a5489af2f33, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:15,177 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727775176"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1684727775176"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1684727775176"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1684727775176"}]},"ts":"1684727775176"} 2023-05-22 03:56:15,180 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-05-22 03:56:15,181 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 3b5b921e3e20a943a4962a5489af2f33, server=jenkins-hbase20.apache.org,40647,1684727774227 in 174 msec 2023-05-22 03:56:15,183 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-05-22 03:56:15,183 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=3b5b921e3e20a943a4962a5489af2f33, ASSIGN in 335 msec 2023-05-22 03:56:15,184 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-05-22 03:56:15,184 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727775184"}]},"ts":"1684727775184"} 2023-05-22 03:56:15,185 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-05-22 03:56:15,188 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-05-22 03:56:15,190 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 393 msec 2023-05-22 03:56:15,198 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-05-22 03:56:15,203 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:56:15,203 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:56:15,207 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-05-22 03:56:15,217 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:56:15,221 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 13 msec 2023-05-22 03:56:15,231 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-05-22 03:56:15,239 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:56:15,243 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 12 msec 2023-05-22 03:56:15,254 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-05-22 03:56:15,256 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-05-22 03:56:15,256 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 1.010sec 2023-05-22 03:56:15,256 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-05-22 03:56:15,256 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-05-22 03:56:15,256 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-05-22 03:56:15,256 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,39195,1684727774188-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-05-22 03:56:15,257 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,39195,1684727774188-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-05-22 03:56:15,258 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-05-22 03:56:15,343 DEBUG [Listener at localhost.localdomain/35753] zookeeper.ReadOnlyZKClient(139): Connect 0x480a525f to 127.0.0.1:58947 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:56:15,348 DEBUG [Listener at localhost.localdomain/35753] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b3539f4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:56:15,353 DEBUG [hconnection-0x34a090fe-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-05-22 03:56:15,356 INFO [RS-EventLoopGroup-11-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50090, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-05-22 03:56:15,358 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,39195,1684727774188 2023-05-22 03:56:15,358 INFO [Listener at localhost.localdomain/35753] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:56:15,362 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-05-22 03:56:15,362 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:56:15,363 INFO [Listener at localhost.localdomain/35753] master.MasterRpcServices(492): Client=null/null set balanceSwitch=false 2023-05-22 03:56:15,365 DEBUG [Listener at localhost.localdomain/35753] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-05-22 03:56:15,369 INFO [RS-EventLoopGroup-10-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:33758, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-05-22 03:56:15,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] util.TableDescriptorChecker(340): MAX_FILESIZE for table descriptor or "hbase.hregion.max.filesize" (786432) is too small, which might cause over splitting into unmanageable number of regions. 2023-05-22 03:56:15,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (8192) is too small, which might cause very frequent flushing. 2023-05-22 03:56:15,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestLogRolling-testCompactionRecordDoesntBlockRolling', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-05-22 03:56:15,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:15,375 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestLogRolling-testCompactionRecordDoesntBlockRolling execute state=CREATE_TABLE_PRE_OPERATION 2023-05-22 03:56:15,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestLogRolling-testCompactionRecordDoesntBlockRolling" procId is: 9 2023-05-22 03:56:15,376 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestLogRolling-testCompactionRecordDoesntBlockRolling execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-05-22 03:56:15,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-05-22 03:56:15,378 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/.tmp/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9 2023-05-22 03:56:15,378 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/.tmp/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9 empty. 2023-05-22 03:56:15,379 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/.tmp/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9 2023-05-22 03:56:15,379 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestLogRolling-testCompactionRecordDoesntBlockRolling regions 2023-05-22 03:56:15,399 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/.tmp/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/.tabledesc/.tableinfo.0000000001 2023-05-22 03:56:15,401 INFO [RegionOpenAndInit-TestLogRolling-testCompactionRecordDoesntBlockRolling-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9c90f192e3d0f5666b140bf77bd4b0c9, NAME => 'TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestLogRolling-testCompactionRecordDoesntBlockRolling', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/.tmp 2023-05-22 03:56:15,408 DEBUG [RegionOpenAndInit-TestLogRolling-testCompactionRecordDoesntBlockRolling-pool-0] regionserver.HRegion(866): Instantiated TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:56:15,409 DEBUG [RegionOpenAndInit-TestLogRolling-testCompactionRecordDoesntBlockRolling-pool-0] regionserver.HRegion(1604): Closing 9c90f192e3d0f5666b140bf77bd4b0c9, disabling compactions & flushes 2023-05-22 03:56:15,409 INFO [RegionOpenAndInit-TestLogRolling-testCompactionRecordDoesntBlockRolling-pool-0] regionserver.HRegion(1626): Closing region TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:56:15,409 DEBUG [RegionOpenAndInit-TestLogRolling-testCompactionRecordDoesntBlockRolling-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:56:15,409 DEBUG [RegionOpenAndInit-TestLogRolling-testCompactionRecordDoesntBlockRolling-pool-0] regionserver.HRegion(1714): Acquired close lock on TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. after waiting 0 ms 2023-05-22 03:56:15,409 DEBUG [RegionOpenAndInit-TestLogRolling-testCompactionRecordDoesntBlockRolling-pool-0] regionserver.HRegion(1724): Updates disabled for region TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:56:15,409 INFO [RegionOpenAndInit-TestLogRolling-testCompactionRecordDoesntBlockRolling-pool-0] regionserver.HRegion(1838): Closed TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:56:15,409 DEBUG [RegionOpenAndInit-TestLogRolling-testCompactionRecordDoesntBlockRolling-pool-0] regionserver.HRegion(1558): Region close journal for 9c90f192e3d0f5666b140bf77bd4b0c9: 2023-05-22 03:56:15,411 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestLogRolling-testCompactionRecordDoesntBlockRolling execute state=CREATE_TABLE_ADD_TO_META 2023-05-22 03:56:15,412 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9.","families":{"info":[{"qualifier":"regioninfo","vlen":87,"tag":[],"timestamp":"1684727775412"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1684727775412"}]},"ts":"1684727775412"} 2023-05-22 03:56:15,413 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-05-22 03:56:15,414 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestLogRolling-testCompactionRecordDoesntBlockRolling execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-05-22 03:56:15,415 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestLogRolling-testCompactionRecordDoesntBlockRolling","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727775415"}]},"ts":"1684727775415"} 2023-05-22 03:56:15,416 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestLogRolling-testCompactionRecordDoesntBlockRolling, state=ENABLING in hbase:meta 2023-05-22 03:56:15,419 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestLogRolling-testCompactionRecordDoesntBlockRolling, region=9c90f192e3d0f5666b140bf77bd4b0c9, ASSIGN}] 2023-05-22 03:56:15,421 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestLogRolling-testCompactionRecordDoesntBlockRolling, region=9c90f192e3d0f5666b140bf77bd4b0c9, ASSIGN 2023-05-22 03:56:15,422 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestLogRolling-testCompactionRecordDoesntBlockRolling, region=9c90f192e3d0f5666b140bf77bd4b0c9, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,40647,1684727774227; forceNewPlan=false, retain=false 2023-05-22 03:56:15,574 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=9c90f192e3d0f5666b140bf77bd4b0c9, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:15,575 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9.","families":{"info":[{"qualifier":"regioninfo","vlen":87,"tag":[],"timestamp":"1684727775574"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1684727775574"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1684727775574"}]},"ts":"1684727775574"} 2023-05-22 03:56:15,580 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 9c90f192e3d0f5666b140bf77bd4b0c9, server=jenkins-hbase20.apache.org,40647,1684727774227}] 2023-05-22 03:56:15,746 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:56:15,746 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9c90f192e3d0f5666b140bf77bd4b0c9, NAME => 'TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:56:15,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestLogRolling-testCompactionRecordDoesntBlockRolling 9c90f192e3d0f5666b140bf77bd4b0c9 2023-05-22 03:56:15,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:56:15,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 9c90f192e3d0f5666b140bf77bd4b0c9 2023-05-22 03:56:15,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 9c90f192e3d0f5666b140bf77bd4b0c9 2023-05-22 03:56:15,749 INFO [StoreOpener-9c90f192e3d0f5666b140bf77bd4b0c9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 9c90f192e3d0f5666b140bf77bd4b0c9 2023-05-22 03:56:15,752 DEBUG [StoreOpener-9c90f192e3d0f5666b140bf77bd4b0c9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info 2023-05-22 03:56:15,752 DEBUG [StoreOpener-9c90f192e3d0f5666b140bf77bd4b0c9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info 2023-05-22 03:56:15,752 INFO [StoreOpener-9c90f192e3d0f5666b140bf77bd4b0c9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9c90f192e3d0f5666b140bf77bd4b0c9 columnFamilyName info 2023-05-22 03:56:15,753 INFO [StoreOpener-9c90f192e3d0f5666b140bf77bd4b0c9-1] regionserver.HStore(310): Store=9c90f192e3d0f5666b140bf77bd4b0c9/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:56:15,754 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9 2023-05-22 03:56:15,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9 2023-05-22 03:56:15,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 9c90f192e3d0f5666b140bf77bd4b0c9 2023-05-22 03:56:15,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:56:15,761 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 9c90f192e3d0f5666b140bf77bd4b0c9; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=783672, jitterRate=-0.003510773181915283}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:56:15,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 9c90f192e3d0f5666b140bf77bd4b0c9: 2023-05-22 03:56:15,762 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9., pid=11, masterSystemTime=1684727775737 2023-05-22 03:56:15,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:56:15,764 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:56:15,764 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=9c90f192e3d0f5666b140bf77bd4b0c9, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:15,765 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9.","families":{"info":[{"qualifier":"regioninfo","vlen":87,"tag":[],"timestamp":"1684727775764"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1684727775764"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1684727775764"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1684727775764"}]},"ts":"1684727775764"} 2023-05-22 03:56:15,769 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-05-22 03:56:15,769 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 9c90f192e3d0f5666b140bf77bd4b0c9, server=jenkins-hbase20.apache.org,40647,1684727774227 in 186 msec 2023-05-22 03:56:15,771 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-05-22 03:56:15,771 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestLogRolling-testCompactionRecordDoesntBlockRolling, region=9c90f192e3d0f5666b140bf77bd4b0c9, ASSIGN in 350 msec 2023-05-22 03:56:15,772 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestLogRolling-testCompactionRecordDoesntBlockRolling execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-05-22 03:56:15,772 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestLogRolling-testCompactionRecordDoesntBlockRolling","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727775772"}]},"ts":"1684727775772"} 2023-05-22 03:56:15,774 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestLogRolling-testCompactionRecordDoesntBlockRolling, state=ENABLED in hbase:meta 2023-05-22 03:56:15,777 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestLogRolling-testCompactionRecordDoesntBlockRolling execute state=CREATE_TABLE_POST_OPERATION 2023-05-22 03:56:15,779 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestLogRolling-testCompactionRecordDoesntBlockRolling in 406 msec 2023-05-22 03:56:18,282 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-05-22 03:56:20,489 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-05-22 03:56:20,491 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-05-22 03:56:20,492 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestLogRolling-testCompactionRecordDoesntBlockRolling' 2023-05-22 03:56:25,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-05-22 03:56:25,380 INFO [Listener at localhost.localdomain/35753] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestLogRolling-testCompactionRecordDoesntBlockRolling, procId: 9 completed 2023-05-22 03:56:25,386 DEBUG [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(2627): Found 1 regions for table TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:25,386 DEBUG [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(2633): firstRegionName=TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:56:25,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-05-22 03:56:25,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] procedure.ProcedureCoordinator(165): Submitting procedure hbase:namespace 2023-05-22 03:56:25,407 INFO [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'hbase:namespace' 2023-05-22 03:56:25,407 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-05-22 03:56:25,407 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'hbase:namespace' starting 'acquire' 2023-05-22 03:56:25,407 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'hbase:namespace', kicking off acquire phase on members. 2023-05-22 03:56:25,408 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/hbase:namespace 2023-05-22 03:56:25,408 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/hbase:namespace 2023-05-22 03:56:25,409 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-05-22 03:56:25,409 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/hbase:namespace/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-05-22 03:56:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:56:25,410 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/hbase:namespace/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,410 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-05-22 03:56:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/hbase:namespace 2023-05-22 03:56:25,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/hbase:namespace 2023-05-22 03:56:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-05-22 03:56:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/hbase:namespace 2023-05-22 03:56:25,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for hbase:namespace 2023-05-22 03:56:25,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:hbase:namespace 2023-05-22 03:56:25,413 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'hbase:namespace' with timeout 60000ms 2023-05-22 03:56:25,413 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-05-22 03:56:25,414 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'hbase:namespace' starting 'acquire' stage 2023-05-22 03:56:25,414 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-05-22 03:56:25,414 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-05-22 03:56:25,414 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. 2023-05-22 03:56:25,415 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. started... 2023-05-22 03:56:25,415 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b5b921e3e20a943a4962a5489af2f33 1/1 column families, dataSize=78 B heapSize=488 B 2023-05-22 03:56:25,428 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/namespace/3b5b921e3e20a943a4962a5489af2f33/.tmp/info/5f9dc3b635574faeaf84f586c283f9f5 2023-05-22 03:56:25,439 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/namespace/3b5b921e3e20a943a4962a5489af2f33/.tmp/info/5f9dc3b635574faeaf84f586c283f9f5 as hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/namespace/3b5b921e3e20a943a4962a5489af2f33/info/5f9dc3b635574faeaf84f586c283f9f5 2023-05-22 03:56:25,447 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/namespace/3b5b921e3e20a943a4962a5489af2f33/info/5f9dc3b635574faeaf84f586c283f9f5, entries=2, sequenceid=6, filesize=4.8 K 2023-05-22 03:56:25,448 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 3b5b921e3e20a943a4962a5489af2f33 in 33ms, sequenceid=6, compaction requested=false 2023-05-22 03:56:25,449 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b5b921e3e20a943a4962a5489af2f33: 2023-05-22 03:56:25,449 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. 2023-05-22 03:56:25,449 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-05-22 03:56:25,450 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-05-22 03:56:25,450 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,450 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'hbase:namespace' locally acquired 2023-05-22 03:56:25,450 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,40647,1684727774227' joining acquired barrier for procedure (hbase:namespace) in zk 2023-05-22 03:56:25,452 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/hbase:namespace/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,452 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/hbase:namespace 2023-05-22 03:56:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/hbase:namespace/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-05-22 03:56:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-05-22 03:56:25,453 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/hbase:namespace 2023-05-22 03:56:25,453 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'hbase:namespace' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-05-22 03:56:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-05-22 03:56:25,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-05-22 03:56:25,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----hbase:namespace 2023-05-22 03:56:25,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-05-22 03:56:25,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,40647,1684727774227' joining acquired barrier for procedure 'hbase:namespace' on coordinator 2023-05-22 03:56:25,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1938b4b8[Count = 0] remaining members to acquire global barrier 2023-05-22 03:56:25,457 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'hbase:namespace' starting 'in-barrier' execution. 2023-05-22 03:56:25,457 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/hbase:namespace 2023-05-22 03:56:25,463 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/hbase:namespace 2023-05-22 03:56:25,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/hbase:namespace 2023-05-22 03:56:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/hbase:namespace 2023-05-22 03:56:25,463 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'hbase:namespace' received 'reached' from coordinator. 2023-05-22 03:56:25,463 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'hbase:namespace' locally completed 2023-05-22 03:56:25,463 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'hbase:namespace' completed for member 'jenkins-hbase20.apache.org,40647,1684727774227' in zk 2023-05-22 03:56:25,463 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/hbase:namespace/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,464 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-05-22 03:56:25,466 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/hbase:namespace/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,466 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'hbase:namespace' has notified controller of completion 2023-05-22 03:56:25,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/hbase:namespace/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-05-22 03:56:25,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-05-22 03:56:25,466 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-05-22 03:56:25,466 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'hbase:namespace' completed. 2023-05-22 03:56:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-05-22 03:56:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-05-22 03:56:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----hbase:namespace 2023-05-22 03:56:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-05-22 03:56:25,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----hbase:namespace 2023-05-22 03:56:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'hbase:namespace' member 'jenkins-hbase20.apache.org,40647,1684727774227': 2023-05-22 03:56:25,473 INFO [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'hbase:namespace' execution completed 2023-05-22 03:56:25,473 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-05-22 03:56:25,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,40647,1684727774227' released barrier for procedure'hbase:namespace', counting down latch. Waiting for 0 more 2023-05-22 03:56:25,473 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-05-22 03:56:25,473 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:hbase:namespace 2023-05-22 03:56:25,473 INFO [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure hbase:namespaceincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-05-22 03:56:25,480 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/hbase:namespace 2023-05-22 03:56:25,480 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/hbase:namespace 2023-05-22 03:56:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/hbase:namespace 2023-05-22 03:56:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-05-22 03:56:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-05-22 03:56:25,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/hbase:namespace 2023-05-22 03:56:25,480 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-05-22 03:56:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/hbase:namespace 2023-05-22 03:56:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-05-22 03:56:25,481 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/hbase:namespace/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-05-22 03:56:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:56:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----hbase:namespace 2023-05-22 03:56:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-05-22 03:56:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/hbase:namespace 2023-05-22 03:56:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----hbase:namespace 2023-05-22 03:56:25,484 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/hbase:namespace/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-05-22 03:56:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----hbase:namespace 2023-05-22 03:56:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,494 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/hbase:namespace/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,494 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/hbase:namespace 2023-05-22 03:56:25,494 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/hbase:namespace 2023-05-22 03:56:25,494 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-05-22 03:56:25,494 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-05-22 03:56:25,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-05-22 03:56:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:56:25,494 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/hbase:namespace/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:25,495 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/hbase:namespace 2023-05-22 03:56:25,495 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/hbase:namespace 2023-05-22 03:56:25,495 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/hbase:namespace 2023-05-22 03:56:25,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-05-22 03:56:25,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:56:25,496 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-05-22 03:56:25,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'hbase:namespace' 2023-05-22 03:56:25,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-05-22 03:56:25,500 DEBUG [Listener at localhost.localdomain/35753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : hbase:namespace'' to complete. (max 20000 ms per retry) 2023-05-22 03:56:25,501 DEBUG [Listener at localhost.localdomain/35753] client.HBaseAdmin(2698): (#1) Sleeping: 10000ms while waiting for procedure completion. 2023-05-22 03:56:35,501 DEBUG [Listener at localhost.localdomain/35753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-05-22 03:56:35,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-05-22 03:56:35,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-05-22 03:56:35,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] procedure.ProcedureCoordinator(165): Submitting procedure TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,523 INFO [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' 2023-05-22 03:56:35,523 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-05-22 03:56:35,524 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' starting 'acquire' 2023-05-22 03:56:35,524 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling', kicking off acquire phase on members. 2023-05-22 03:56:35,525 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,525 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,526 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-05-22 03:56:35,526 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-05-22 03:56:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:56:35,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,527 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,527 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-05-22 03:56:35,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-05-22 03:56:35,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,528 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' with timeout 60000ms 2023-05-22 03:56:35,528 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-05-22 03:56:35,529 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' starting 'acquire' stage 2023-05-22 03:56:35,529 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-05-22 03:56:35,529 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-05-22 03:56:35,529 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:56:35,529 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. started... 2023-05-22 03:56:35,530 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9c90f192e3d0f5666b140bf77bd4b0c9 1/1 column families, dataSize=1.05 KB heapSize=1.38 KB 2023-05-22 03:56:35,546 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.05 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/.tmp/info/3ef46768f9764c8b9d0a384dc05c40be 2023-05-22 03:56:35,556 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/.tmp/info/3ef46768f9764c8b9d0a384dc05c40be as hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/3ef46768f9764c8b9d0a384dc05c40be 2023-05-22 03:56:35,564 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/3ef46768f9764c8b9d0a384dc05c40be, entries=1, sequenceid=5, filesize=5.8 K 2023-05-22 03:56:35,565 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.05 KB/1076, heapSize ~1.36 KB/1392, currentSize=0 B/0 for 9c90f192e3d0f5666b140bf77bd4b0c9 in 35ms, sequenceid=5, compaction requested=false 2023-05-22 03:56:35,566 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9c90f192e3d0f5666b140bf77bd4b0c9: 2023-05-22 03:56:35,566 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:56:35,566 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-05-22 03:56:35,566 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-05-22 03:56:35,566 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,566 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' locally acquired 2023-05-22 03:56:35,566 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,40647,1684727774227' joining acquired barrier for procedure (TestLogRolling-testCompactionRecordDoesntBlockRolling) in zk 2023-05-22 03:56:35,568 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,568 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-05-22 03:56:35,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-05-22 03:56:35,568 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,569 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-05-22 03:56:35,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-05-22 03:56:35,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-05-22 03:56:35,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-05-22 03:56:35,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,40647,1684727774227' joining acquired barrier for procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' on coordinator 2023-05-22 03:56:35,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d073347[Count = 0] remaining members to acquire global barrier 2023-05-22 03:56:35,570 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' starting 'in-barrier' execution. 2023-05-22 03:56:35,570 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,570 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,571 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' received 'reached' from coordinator. 2023-05-22 03:56:35,571 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' locally completed 2023-05-22 03:56:35,571 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' completed for member 'jenkins-hbase20.apache.org,40647,1684727774227' in zk 2023-05-22 03:56:35,571 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,571 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-05-22 03:56:35,572 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' has notified controller of completion 2023-05-22 03:56:35,572 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-05-22 03:56:35,572 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,572 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' completed. 2023-05-22 03:56:35,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-05-22 03:56:35,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-05-22 03:56:35,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-05-22 03:56:35,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-05-22 03:56:35,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-05-22 03:56:35,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' member 'jenkins-hbase20.apache.org,40647,1684727774227': 2023-05-22 03:56:35,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,40647,1684727774227' released barrier for procedure'TestLogRolling-testCompactionRecordDoesntBlockRolling', counting down latch. Waiting for 0 more 2023-05-22 03:56:35,576 INFO [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' execution completed 2023-05-22 03:56:35,576 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-05-22 03:56:35,576 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-05-22 03:56:35,576 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,576 INFO [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestLogRolling-testCompactionRecordDoesntBlockRollingincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-05-22 03:56:35,580 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,580 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-05-22 03:56:35,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-05-22 03:56:35,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,580 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-05-22 03:56:35,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-05-22 03:56:35,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-05-22 03:56:35,580 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:56:35,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-05-22 03:56:35,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,582 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-05-22 03:56:35,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,587 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,587 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-05-22 03:56:35,587 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-05-22 03:56:35,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestLogRolling-testCompactionRecordDoesntBlockRolling' 2023-05-22 03:56:35,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-05-22 03:56:35,587 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-05-22 03:56:35,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:56:35,588 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-05-22 03:56:35,588 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,589 DEBUG [Listener at localhost.localdomain/35753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestLogRolling-testCompactionRecordDoesntBlockRolling'' to complete. (max 20000 ms per retry) 2023-05-22 03:56:35,589 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:35,589 DEBUG [Listener at localhost.localdomain/35753] client.HBaseAdmin(2698): (#1) Sleeping: 10000ms while waiting for procedure completion. 2023-05-22 03:56:35,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-05-22 03:56:35,589 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,589 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,589 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:35,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:56:45,589 DEBUG [Listener at localhost.localdomain/35753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-05-22 03:56:45,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-05-22 03:56:45,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-05-22 03:56:45,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] procedure.ProcedureCoordinator(143): Procedure TestLogRolling-testCompactionRecordDoesntBlockRolling was in running list but was completed. Accepting new attempt. 2023-05-22 03:56:45,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] procedure.ProcedureCoordinator(165): Submitting procedure TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,607 INFO [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' 2023-05-22 03:56:45,607 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-05-22 03:56:45,608 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' starting 'acquire' 2023-05-22 03:56:45,608 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling', kicking off acquire phase on members. 2023-05-22 03:56:45,608 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,608 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,609 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-05-22 03:56:45,609 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-05-22 03:56:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:56:45,609 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,609 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-05-22 03:56:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-05-22 03:56:45,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestLogRolling-testCompactionRecordDoesntBlockRolling is still present, removing 2023-05-22 03:56:45,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,610 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' with timeout 60000ms 2023-05-22 03:56:45,610 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-05-22 03:56:45,611 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' starting 'acquire' stage 2023-05-22 03:56:45,611 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-05-22 03:56:45,611 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-05-22 03:56:45,611 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:56:45,611 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. started... 2023-05-22 03:56:45,611 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9c90f192e3d0f5666b140bf77bd4b0c9 1/1 column families, dataSize=1.05 KB heapSize=1.38 KB 2023-05-22 03:56:45,622 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.05 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/.tmp/info/f43a0e109fe545a2913df5434f71ddd8 2023-05-22 03:56:45,629 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/.tmp/info/f43a0e109fe545a2913df5434f71ddd8 as hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/f43a0e109fe545a2913df5434f71ddd8 2023-05-22 03:56:45,637 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/f43a0e109fe545a2913df5434f71ddd8, entries=1, sequenceid=9, filesize=5.8 K 2023-05-22 03:56:45,639 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.05 KB/1076, heapSize ~1.36 KB/1392, currentSize=0 B/0 for 9c90f192e3d0f5666b140bf77bd4b0c9 in 28ms, sequenceid=9, compaction requested=false 2023-05-22 03:56:45,639 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9c90f192e3d0f5666b140bf77bd4b0c9: 2023-05-22 03:56:45,639 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:56:45,639 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-05-22 03:56:45,639 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-05-22 03:56:45,639 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,639 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' locally acquired 2023-05-22 03:56:45,639 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,40647,1684727774227' joining acquired barrier for procedure (TestLogRolling-testCompactionRecordDoesntBlockRolling) in zk 2023-05-22 03:56:45,641 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,641 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-05-22 03:56:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-05-22 03:56:45,642 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,642 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-05-22 03:56:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-05-22 03:56:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-05-22 03:56:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-05-22 03:56:45,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,40647,1684727774227' joining acquired barrier for procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' on coordinator 2023-05-22 03:56:45,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5805be83[Count = 0] remaining members to acquire global barrier 2023-05-22 03:56:45,644 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' starting 'in-barrier' execution. 2023-05-22 03:56:45,644 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,645 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,645 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' received 'reached' from coordinator. 2023-05-22 03:56:45,645 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' locally completed 2023-05-22 03:56:45,645 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' completed for member 'jenkins-hbase20.apache.org,40647,1684727774227' in zk 2023-05-22 03:56:45,645 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,646 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-05-22 03:56:45,647 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' has notified controller of completion 2023-05-22 03:56:45,647 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,647 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-05-22 03:56:45,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-05-22 03:56:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-05-22 03:56:45,647 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' completed. 2023-05-22 03:56:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-05-22 03:56:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-05-22 03:56:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-05-22 03:56:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' member 'jenkins-hbase20.apache.org,40647,1684727774227': 2023-05-22 03:56:45,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,40647,1684727774227' released barrier for procedure'TestLogRolling-testCompactionRecordDoesntBlockRolling', counting down latch. Waiting for 0 more 2023-05-22 03:56:45,651 INFO [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' execution completed 2023-05-22 03:56:45,651 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-05-22 03:56:45,651 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-05-22 03:56:45,651 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,651 INFO [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestLogRolling-testCompactionRecordDoesntBlockRollingincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-05-22 03:56:45,653 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,654 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,654 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-05-22 03:56:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-05-22 03:56:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-05-22 03:56:45,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-05-22 03:56:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-05-22 03:56:45,654 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:56:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-05-22 03:56:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,656 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-05-22 03:56:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,668 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,668 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-05-22 03:56:45,668 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,668 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestLogRolling-testCompactionRecordDoesntBlockRolling' 2023-05-22 03:56:45,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-05-22 03:56:45,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-05-22 03:56:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:56:45,668 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-05-22 03:56:45,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling because node does not exist (not an error) 2023-05-22 03:56:45,668 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:45,668 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-05-22 03:56:45,669 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,669 DEBUG [Listener at localhost.localdomain/35753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestLogRolling-testCompactionRecordDoesntBlockRolling'' to complete. (max 20000 ms per retry) 2023-05-22 03:56:45,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-05-22 03:56:45,669 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:56:45,670 DEBUG [Listener at localhost.localdomain/35753] client.HBaseAdmin(2698): (#1) Sleeping: 10000ms while waiting for procedure completion. 2023-05-22 03:56:45,670 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,670 DEBUG [Listener at localhost.localdomain/35753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-05-22 03:56:55,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-05-22 03:56:55,687 INFO [Listener at localhost.localdomain/35753] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227/jenkins-hbase20.apache.org%2C40647%2C1684727774227.1684727774622 with entries=13, filesize=6.44 KB; new WAL /user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227/jenkins-hbase20.apache.org%2C40647%2C1684727774227.1684727815675 2023-05-22 03:56:55,688 DEBUG [Listener at localhost.localdomain/35753] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46309,DS-9e3e0a3e-2325-44be-89a8-4152a9f276f7,DISK], DatanodeInfoWithStorage[127.0.0.1:42319,DS-2e428832-7c1d-4bb3-a716-f1edebf826ea,DISK]] 2023-05-22 03:56:55,688 DEBUG [Listener at localhost.localdomain/35753] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227/jenkins-hbase20.apache.org%2C40647%2C1684727774227.1684727774622 is not closed yet, will try archiving it next time 2023-05-22 03:56:55,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-05-22 03:56:55,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] procedure.ProcedureCoordinator(143): Procedure TestLogRolling-testCompactionRecordDoesntBlockRolling was in running list but was completed. Accepting new attempt. 2023-05-22 03:56:55,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] procedure.ProcedureCoordinator(165): Submitting procedure TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,696 INFO [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' 2023-05-22 03:56:55,696 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-05-22 03:56:55,697 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' starting 'acquire' 2023-05-22 03:56:55,697 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling', kicking off acquire phase on members. 2023-05-22 03:56:55,697 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,697 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,698 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,698 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-05-22 03:56:55,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-05-22 03:56:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:56:55,698 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,698 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-05-22 03:56:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-05-22 03:56:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestLogRolling-testCompactionRecordDoesntBlockRolling is still present, removing 2023-05-22 03:56:55,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,700 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' with timeout 60000ms 2023-05-22 03:56:55,700 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-05-22 03:56:55,700 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' starting 'acquire' stage 2023-05-22 03:56:55,701 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-05-22 03:56:55,701 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-05-22 03:56:55,701 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:56:55,701 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. started... 2023-05-22 03:56:55,701 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9c90f192e3d0f5666b140bf77bd4b0c9 1/1 column families, dataSize=1.05 KB heapSize=1.38 KB 2023-05-22 03:56:55,713 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.05 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/.tmp/info/69f9ad34f9b44e72af3710b7f42e4254 2023-05-22 03:56:55,719 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/.tmp/info/69f9ad34f9b44e72af3710b7f42e4254 as hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/69f9ad34f9b44e72af3710b7f42e4254 2023-05-22 03:56:55,724 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/69f9ad34f9b44e72af3710b7f42e4254, entries=1, sequenceid=13, filesize=5.8 K 2023-05-22 03:56:55,725 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.05 KB/1076, heapSize ~1.36 KB/1392, currentSize=0 B/0 for 9c90f192e3d0f5666b140bf77bd4b0c9 in 24ms, sequenceid=13, compaction requested=true 2023-05-22 03:56:55,725 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9c90f192e3d0f5666b140bf77bd4b0c9: 2023-05-22 03:56:55,725 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:56:55,725 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-05-22 03:56:55,725 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-05-22 03:56:55,725 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,725 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' locally acquired 2023-05-22 03:56:55,726 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,40647,1684727774227' joining acquired barrier for procedure (TestLogRolling-testCompactionRecordDoesntBlockRolling) in zk 2023-05-22 03:56:55,727 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,727 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-05-22 03:56:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-05-22 03:56:55,728 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,728 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-05-22 03:56:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-05-22 03:56:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-05-22 03:56:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-05-22 03:56:55,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,40647,1684727774227' joining acquired barrier for procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' on coordinator 2023-05-22 03:56:55,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35566b54[Count = 0] remaining members to acquire global barrier 2023-05-22 03:56:55,729 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' starting 'in-barrier' execution. 2023-05-22 03:56:55,729 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,729 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,730 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' received 'reached' from coordinator. 2023-05-22 03:56:55,730 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' locally completed 2023-05-22 03:56:55,730 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,730 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-05-22 03:56:55,730 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' completed for member 'jenkins-hbase20.apache.org,40647,1684727774227' in zk 2023-05-22 03:56:55,731 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' has notified controller of completion 2023-05-22 03:56:55,731 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,731 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-05-22 03:56:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-05-22 03:56:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-05-22 03:56:55,731 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' completed. 2023-05-22 03:56:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-05-22 03:56:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-05-22 03:56:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-05-22 03:56:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' member 'jenkins-hbase20.apache.org,40647,1684727774227': 2023-05-22 03:56:55,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,40647,1684727774227' released barrier for procedure'TestLogRolling-testCompactionRecordDoesntBlockRolling', counting down latch. Waiting for 0 more 2023-05-22 03:56:55,734 INFO [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' execution completed 2023-05-22 03:56:55,734 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-05-22 03:56:55,734 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-05-22 03:56:55,734 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,734 INFO [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestLogRolling-testCompactionRecordDoesntBlockRollingincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-05-22 03:56:55,735 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,735 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-05-22 03:56:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-05-22 03:56:55,735 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-05-22 03:56:55,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-05-22 03:56:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:56:55,736 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-05-22 03:56:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-05-22 03:56:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,737 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-05-22 03:56:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,740 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,740 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-05-22 03:56:55,740 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-05-22 03:56:55,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-05-22 03:56:55,740 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:56:55,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestLogRolling-testCompactionRecordDoesntBlockRolling' 2023-05-22 03:56:55,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-05-22 03:56:55,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling because node does not exist (not an error) 2023-05-22 03:56:55,740 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-05-22 03:56:55,741 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,741 DEBUG [Listener at localhost.localdomain/35753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestLogRolling-testCompactionRecordDoesntBlockRolling'' to complete. (max 20000 ms per retry) 2023-05-22 03:56:55,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-05-22 03:56:55,741 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:56:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:56:55,741 DEBUG [Listener at localhost.localdomain/35753] client.HBaseAdmin(2698): (#1) Sleeping: 10000ms while waiting for procedure completion. 2023-05-22 03:56:55,741 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,741 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:56:55,741 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,741 DEBUG [Listener at localhost.localdomain/35753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-05-22 03:57:05,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-05-22 03:57:05,744 DEBUG [Listener at localhost.localdomain/35753] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-05-22 03:57:05,754 DEBUG [Listener at localhost.localdomain/35753] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 17769 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-05-22 03:57:05,754 DEBUG [Listener at localhost.localdomain/35753] regionserver.HStore(1912): 9c90f192e3d0f5666b140bf77bd4b0c9/info is initiating minor compaction (all files) 2023-05-22 03:57:05,754 INFO [Listener at localhost.localdomain/35753] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-05-22 03:57:05,754 INFO [Listener at localhost.localdomain/35753] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:57:05,755 INFO [Listener at localhost.localdomain/35753] regionserver.HRegion(2259): Starting compaction of 9c90f192e3d0f5666b140bf77bd4b0c9/info in TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:57:05,756 INFO [Listener at localhost.localdomain/35753] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/3ef46768f9764c8b9d0a384dc05c40be, hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/f43a0e109fe545a2913df5434f71ddd8, hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/69f9ad34f9b44e72af3710b7f42e4254] into tmpdir=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/.tmp, totalSize=17.4 K 2023-05-22 03:57:05,756 DEBUG [Listener at localhost.localdomain/35753] compactions.Compactor(207): Compacting 3ef46768f9764c8b9d0a384dc05c40be, keycount=1, bloomtype=ROW, size=5.8 K, encoding=NONE, compression=NONE, seqNum=5, earliestPutTs=1684727795516 2023-05-22 03:57:05,757 DEBUG [Listener at localhost.localdomain/35753] compactions.Compactor(207): Compacting f43a0e109fe545a2913df5434f71ddd8, keycount=1, bloomtype=ROW, size=5.8 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1684727805593 2023-05-22 03:57:05,758 DEBUG [Listener at localhost.localdomain/35753] compactions.Compactor(207): Compacting 69f9ad34f9b44e72af3710b7f42e4254, keycount=1, bloomtype=ROW, size=5.8 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1684727815673 2023-05-22 03:57:05,778 INFO [Listener at localhost.localdomain/35753] throttle.PressureAwareThroughputController(145): 9c90f192e3d0f5666b140bf77bd4b0c9#info#compaction#20 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-05-22 03:57:05,792 DEBUG [Listener at localhost.localdomain/35753] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/.tmp/info/a802d6acbb064d24a956144fccc9488c as hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/a802d6acbb064d24a956144fccc9488c 2023-05-22 03:57:05,798 INFO [Listener at localhost.localdomain/35753] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9c90f192e3d0f5666b140bf77bd4b0c9/info of 9c90f192e3d0f5666b140bf77bd4b0c9 into a802d6acbb064d24a956144fccc9488c(size=8.0 K), total size for store is 8.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-05-22 03:57:05,798 DEBUG [Listener at localhost.localdomain/35753] regionserver.HRegion(2289): Compaction status journal for 9c90f192e3d0f5666b140bf77bd4b0c9: 2023-05-22 03:57:05,809 INFO [Listener at localhost.localdomain/35753] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227/jenkins-hbase20.apache.org%2C40647%2C1684727774227.1684727815675 with entries=4, filesize=2.45 KB; new WAL /user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227/jenkins-hbase20.apache.org%2C40647%2C1684727774227.1684727825800 2023-05-22 03:57:05,809 DEBUG [Listener at localhost.localdomain/35753] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46309,DS-9e3e0a3e-2325-44be-89a8-4152a9f276f7,DISK], DatanodeInfoWithStorage[127.0.0.1:42319,DS-2e428832-7c1d-4bb3-a716-f1edebf826ea,DISK]] 2023-05-22 03:57:05,809 DEBUG [Listener at localhost.localdomain/35753] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227/jenkins-hbase20.apache.org%2C40647%2C1684727774227.1684727815675 is not closed yet, will try archiving it next time 2023-05-22 03:57:05,809 INFO [WAL-Archive-0] wal.AbstractFSWAL(783): Archiving hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227/jenkins-hbase20.apache.org%2C40647%2C1684727774227.1684727774622 to hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/oldWALs/jenkins-hbase20.apache.org%2C40647%2C1684727774227.1684727774622 2023-05-22 03:57:05,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-05-22 03:57:05,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] procedure.ProcedureCoordinator(143): Procedure TestLogRolling-testCompactionRecordDoesntBlockRolling was in running list but was completed. Accepting new attempt. 2023-05-22 03:57:05,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] procedure.ProcedureCoordinator(165): Submitting procedure TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,818 INFO [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' 2023-05-22 03:57:05,818 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-05-22 03:57:05,819 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' starting 'acquire' 2023-05-22 03:57:05,819 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling', kicking off acquire phase on members. 2023-05-22 03:57:05,819 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,819 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,821 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-05-22 03:57:05,821 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-05-22 03:57:05,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:57:05,821 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,821 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-05-22 03:57:05,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-05-22 03:57:05,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestLogRolling-testCompactionRecordDoesntBlockRolling is still present, removing 2023-05-22 03:57:05,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,822 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' with timeout 60000ms 2023-05-22 03:57:05,822 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-05-22 03:57:05,823 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' starting 'acquire' stage 2023-05-22 03:57:05,823 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-05-22 03:57:05,823 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-05-22 03:57:05,823 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:57:05,823 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. started... 2023-05-22 03:57:05,823 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9c90f192e3d0f5666b140bf77bd4b0c9 1/1 column families, dataSize=1.05 KB heapSize=1.38 KB 2023-05-22 03:57:05,836 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.05 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/.tmp/info/5fb64ff3d3da41dc840ad33b45f4e41e 2023-05-22 03:57:05,842 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/.tmp/info/5fb64ff3d3da41dc840ad33b45f4e41e as hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/5fb64ff3d3da41dc840ad33b45f4e41e 2023-05-22 03:57:05,847 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/5fb64ff3d3da41dc840ad33b45f4e41e, entries=1, sequenceid=18, filesize=5.8 K 2023-05-22 03:57:05,848 INFO [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.05 KB/1076, heapSize ~1.36 KB/1392, currentSize=0 B/0 for 9c90f192e3d0f5666b140bf77bd4b0c9 in 25ms, sequenceid=18, compaction requested=false 2023-05-22 03:57:05,848 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9c90f192e3d0f5666b140bf77bd4b0c9: 2023-05-22 03:57:05,848 DEBUG [rs(jenkins-hbase20.apache.org,40647,1684727774227)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:57:05,849 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-05-22 03:57:05,849 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-05-22 03:57:05,849 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,849 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' locally acquired 2023-05-22 03:57:05,849 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,40647,1684727774227' joining acquired barrier for procedure (TestLogRolling-testCompactionRecordDoesntBlockRolling) in zk 2023-05-22 03:57:05,850 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,850 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-05-22 03:57:05,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-05-22 03:57:05,851 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,851 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-05-22 03:57:05,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-05-22 03:57:05,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-05-22 03:57:05,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-05-22 03:57:05,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,40647,1684727774227' joining acquired barrier for procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' on coordinator 2023-05-22 03:57:05,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69178432[Count = 0] remaining members to acquire global barrier 2023-05-22 03:57:05,852 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' starting 'in-barrier' execution. 2023-05-22 03:57:05,852 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,853 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,853 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' received 'reached' from coordinator. 2023-05-22 03:57:05,853 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' locally completed 2023-05-22 03:57:05,853 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' completed for member 'jenkins-hbase20.apache.org,40647,1684727774227' in zk 2023-05-22 03:57:05,853 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,853 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-05-22 03:57:05,855 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,855 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' has notified controller of completion 2023-05-22 03:57:05,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-05-22 03:57:05,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-05-22 03:57:05,855 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-05-22 03:57:05,855 DEBUG [member: 'jenkins-hbase20.apache.org,40647,1684727774227' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' completed. 2023-05-22 03:57:05,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-05-22 03:57:05,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-05-22 03:57:05,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-05-22 03:57:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' member 'jenkins-hbase20.apache.org,40647,1684727774227': 2023-05-22 03:57:05,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,40647,1684727774227' released barrier for procedure'TestLogRolling-testCompactionRecordDoesntBlockRolling', counting down latch. Waiting for 0 more 2023-05-22 03:57:05,858 INFO [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestLogRolling-testCompactionRecordDoesntBlockRolling' execution completed 2023-05-22 03:57:05,858 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-05-22 03:57:05,858 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-05-22 03:57:05,858 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,858 INFO [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestLogRolling-testCompactionRecordDoesntBlockRollingincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-05-22 03:57:05,859 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,859 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-05-22 03:57:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-05-22 03:57:05,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,859 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-05-22 03:57:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-05-22 03:57:05,859 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-05-22 03:57:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:57:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-05-22 03:57:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,861 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-05-22 03:57:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,876 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,876 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,876 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-05-22 03:57:05,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestLogRolling-testCompactionRecordDoesntBlockRolling' 2023-05-22 03:57:05,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-05-22 03:57:05,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-05-22 03:57:05,876 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-05-22 03:57:05,876 DEBUG [(jenkins-hbase20.apache.org,39195,1684727774188)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-05-22 03:57:05,876 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,876 DEBUG [Listener at localhost.localdomain/35753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestLogRolling-testCompactionRecordDoesntBlockRolling'' to complete. (max 20000 ms per retry) 2023-05-22 03:57:05,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling because node does not exist (not an error) 2023-05-22 03:57:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:57:05,877 DEBUG [Listener at localhost.localdomain/35753] client.HBaseAdmin(2698): (#1) Sleeping: 10000ms while waiting for procedure completion. 2023-05-22 03:57:05,877 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:05,877 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,877 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,877 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestLogRolling-testCompactionRecordDoesntBlockRolling 2023-05-22 03:57:05,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-05-22 03:57:05,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:57:15,877 DEBUG [Listener at localhost.localdomain/35753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-05-22 03:57:15,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39195] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-05-22 03:57:15,903 INFO [Listener at localhost.localdomain/35753] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227/jenkins-hbase20.apache.org%2C40647%2C1684727774227.1684727825800 with entries=3, filesize=1.97 KB; new WAL /user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227/jenkins-hbase20.apache.org%2C40647%2C1684727774227.1684727835889 2023-05-22 03:57:15,903 DEBUG [Listener at localhost.localdomain/35753] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46309,DS-9e3e0a3e-2325-44be-89a8-4152a9f276f7,DISK], DatanodeInfoWithStorage[127.0.0.1:42319,DS-2e428832-7c1d-4bb3-a716-f1edebf826ea,DISK]] 2023-05-22 03:57:15,903 DEBUG [Listener at localhost.localdomain/35753] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227/jenkins-hbase20.apache.org%2C40647%2C1684727774227.1684727825800 is not closed yet, will try archiving it next time 2023-05-22 03:57:15,903 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-05-22 03:57:15,903 INFO [WAL-Archive-0] wal.AbstractFSWAL(783): Archiving hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227/jenkins-hbase20.apache.org%2C40647%2C1684727774227.1684727815675 to hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/oldWALs/jenkins-hbase20.apache.org%2C40647%2C1684727774227.1684727815675 2023-05-22 03:57:15,904 INFO [Listener at localhost.localdomain/35753] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-05-22 03:57:15,904 DEBUG [Listener at localhost.localdomain/35753] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x480a525f to 127.0.0.1:58947 2023-05-22 03:57:15,904 DEBUG [Listener at localhost.localdomain/35753] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:57:15,904 DEBUG [Listener at localhost.localdomain/35753] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-05-22 03:57:15,905 DEBUG [Listener at localhost.localdomain/35753] util.JVMClusterUtil(257): Found active master hash=520466428, stopped=false 2023-05-22 03:57:15,905 INFO [Listener at localhost.localdomain/35753] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,39195,1684727774188 2023-05-22 03:57:15,908 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-05-22 03:57:15,908 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:57:15,908 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-05-22 03:57:15,908 INFO [Listener at localhost.localdomain/35753] procedure2.ProcedureExecutor(629): Stopping 2023-05-22 03:57:15,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:57:15,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:57:15,909 DEBUG [Listener at localhost.localdomain/35753] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e8b6c66 to 127.0.0.1:58947 2023-05-22 03:57:15,910 DEBUG [Listener at localhost.localdomain/35753] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:57:15,910 INFO [Listener at localhost.localdomain/35753] regionserver.HRegionServer(2295): ***** STOPPING region server 'jenkins-hbase20.apache.org,40647,1684727774227' ***** 2023-05-22 03:57:15,910 INFO [Listener at localhost.localdomain/35753] regionserver.HRegionServer(2309): STOPPED: Shutdown requested 2023-05-22 03:57:15,910 INFO [RS:0;jenkins-hbase20:40647] regionserver.HeapMemoryManager(220): Stopping 2023-05-22 03:57:15,911 INFO [RS:0;jenkins-hbase20:40647] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-05-22 03:57:15,911 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-05-22 03:57:15,911 INFO [RS:0;jenkins-hbase20:40647] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-05-22 03:57:15,911 INFO [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(3303): Received CLOSE for 9c90f192e3d0f5666b140bf77bd4b0c9 2023-05-22 03:57:15,912 INFO [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(3303): Received CLOSE for 3b5b921e3e20a943a4962a5489af2f33 2023-05-22 03:57:15,912 INFO [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:15,912 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 9c90f192e3d0f5666b140bf77bd4b0c9, disabling compactions & flushes 2023-05-22 03:57:15,912 DEBUG [RS:0;jenkins-hbase20:40647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x170edb77 to 127.0.0.1:58947 2023-05-22 03:57:15,912 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:57:15,912 DEBUG [RS:0;jenkins-hbase20:40647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:57:15,912 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:57:15,912 INFO [RS:0;jenkins-hbase20:40647] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-05-22 03:57:15,912 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. after waiting 0 ms 2023-05-22 03:57:15,912 INFO [RS:0;jenkins-hbase20:40647] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-05-22 03:57:15,912 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:57:15,912 INFO [RS:0;jenkins-hbase20:40647] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-05-22 03:57:15,912 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 9c90f192e3d0f5666b140bf77bd4b0c9 1/1 column families, dataSize=1.05 KB heapSize=1.38 KB 2023-05-22 03:57:15,912 INFO [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(3303): Received CLOSE for 1588230740 2023-05-22 03:57:15,912 INFO [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-05-22 03:57:15,913 DEBUG [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(1478): Online Regions={9c90f192e3d0f5666b140bf77bd4b0c9=TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9., 3b5b921e3e20a943a4962a5489af2f33=hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33., 1588230740=hbase:meta,,1.1588230740} 2023-05-22 03:57:15,914 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-05-22 03:57:15,914 DEBUG [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(1504): Waiting on 1588230740, 3b5b921e3e20a943a4962a5489af2f33, 9c90f192e3d0f5666b140bf77bd4b0c9 2023-05-22 03:57:15,914 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-05-22 03:57:15,914 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-05-22 03:57:15,914 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-05-22 03:57:15,914 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-05-22 03:57:15,914 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=3.10 KB heapSize=5.61 KB 2023-05-22 03:57:15,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.05 KB at sequenceid=22 (bloomFilter=true), to=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/.tmp/info/9b711663463e410e9b3296f15391d98e 2023-05-22 03:57:15,939 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.85 KB at sequenceid=14 (bloomFilter=false), to=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/.tmp/info/91c34d21aa1744a5a58aeb0307216f6b 2023-05-22 03:57:15,941 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/.tmp/info/9b711663463e410e9b3296f15391d98e as hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/9b711663463e410e9b3296f15391d98e 2023-05-22 03:57:15,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/9b711663463e410e9b3296f15391d98e, entries=1, sequenceid=22, filesize=5.8 K 2023-05-22 03:57:15,947 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.05 KB/1076, heapSize ~1.36 KB/1392, currentSize=0 B/0 for 9c90f192e3d0f5666b140bf77bd4b0c9 in 35ms, sequenceid=22, compaction requested=true 2023-05-22 03:57:15,951 DEBUG [StoreCloser-TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/3ef46768f9764c8b9d0a384dc05c40be, hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/f43a0e109fe545a2913df5434f71ddd8, hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/69f9ad34f9b44e72af3710b7f42e4254] to archive 2023-05-22 03:57:15,952 DEBUG [StoreCloser-TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-05-22 03:57:15,954 DEBUG [StoreCloser-TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/3ef46768f9764c8b9d0a384dc05c40be to hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/archive/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/3ef46768f9764c8b9d0a384dc05c40be 2023-05-22 03:57:15,955 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=264 B at sequenceid=14 (bloomFilter=false), to=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/.tmp/table/738cf185090548889077f85f66dac609 2023-05-22 03:57:15,956 DEBUG [StoreCloser-TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/f43a0e109fe545a2913df5434f71ddd8 to hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/archive/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/f43a0e109fe545a2913df5434f71ddd8 2023-05-22 03:57:15,957 DEBUG [StoreCloser-TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/69f9ad34f9b44e72af3710b7f42e4254 to hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/archive/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/info/69f9ad34f9b44e72af3710b7f42e4254 2023-05-22 03:57:15,964 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/.tmp/info/91c34d21aa1744a5a58aeb0307216f6b as hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/info/91c34d21aa1744a5a58aeb0307216f6b 2023-05-22 03:57:15,970 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/default/TestLogRolling-testCompactionRecordDoesntBlockRolling/9c90f192e3d0f5666b140bf77bd4b0c9/recovered.edits/25.seqid, newMaxSeqId=25, maxSeqId=1 2023-05-22 03:57:15,970 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/info/91c34d21aa1744a5a58aeb0307216f6b, entries=20, sequenceid=14, filesize=7.6 K 2023-05-22 03:57:15,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:57:15,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 9c90f192e3d0f5666b140bf77bd4b0c9: 2023-05-22 03:57:15,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed TestLogRolling-testCompactionRecordDoesntBlockRolling,,1684727775370.9c90f192e3d0f5666b140bf77bd4b0c9. 2023-05-22 03:57:15,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 3b5b921e3e20a943a4962a5489af2f33, disabling compactions & flushes 2023-05-22 03:57:15,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. 2023-05-22 03:57:15,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. 2023-05-22 03:57:15,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. after waiting 0 ms 2023-05-22 03:57:15,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. 2023-05-22 03:57:15,971 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/.tmp/table/738cf185090548889077f85f66dac609 as hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/table/738cf185090548889077f85f66dac609 2023-05-22 03:57:15,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/namespace/3b5b921e3e20a943a4962a5489af2f33/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-05-22 03:57:15,978 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. 2023-05-22 03:57:15,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 3b5b921e3e20a943a4962a5489af2f33: 2023-05-22 03:57:15,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1684727774795.3b5b921e3e20a943a4962a5489af2f33. 2023-05-22 03:57:15,979 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/table/738cf185090548889077f85f66dac609, entries=4, sequenceid=14, filesize=4.9 K 2023-05-22 03:57:15,980 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.10 KB/3178, heapSize ~5.33 KB/5456, currentSize=0 B/0 for 1588230740 in 66ms, sequenceid=14, compaction requested=false 2023-05-22 03:57:15,986 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/data/hbase/meta/1588230740/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=1 2023-05-22 03:57:15,987 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-05-22 03:57:15,988 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-05-22 03:57:15,988 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-05-22 03:57:15,988 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-05-22 03:57:16,114 INFO [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,40647,1684727774227; all regions closed. 2023-05-22 03:57:16,115 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:16,128 DEBUG [RS:0;jenkins-hbase20:40647] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/oldWALs 2023-05-22 03:57:16,128 INFO [RS:0;jenkins-hbase20:40647] wal.AbstractFSWAL(1031): Closed WAL: FSHLog jenkins-hbase20.apache.org%2C40647%2C1684727774227.meta:.meta(num 1684727774735) 2023-05-22 03:57:16,128 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/WALs/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:16,133 DEBUG [RS:0;jenkins-hbase20:40647] wal.AbstractFSWAL(1028): Moved 2 WAL file(s) to /user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/oldWALs 2023-05-22 03:57:16,133 INFO [RS:0;jenkins-hbase20:40647] wal.AbstractFSWAL(1031): Closed WAL: FSHLog jenkins-hbase20.apache.org%2C40647%2C1684727774227:(num 1684727835889) 2023-05-22 03:57:16,133 DEBUG [RS:0;jenkins-hbase20:40647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:57:16,134 INFO [RS:0;jenkins-hbase20:40647] regionserver.LeaseManager(133): Closed leases 2023-05-22 03:57:16,134 INFO [RS:0;jenkins-hbase20:40647] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-05-22 03:57:16,134 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-05-22 03:57:16,135 INFO [RS:0;jenkins-hbase20:40647] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:40647 2023-05-22 03:57:16,138 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,40647,1684727774227 2023-05-22 03:57:16,138 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:57:16,138 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:57:16,139 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,40647,1684727774227] 2023-05-22 03:57:16,139 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,40647,1684727774227; numProcessing=1 2023-05-22 03:57:16,140 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,40647,1684727774227 already deleted, retry=false 2023-05-22 03:57:16,140 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,40647,1684727774227 expired; onlineServers=0 2023-05-22 03:57:16,140 INFO [RegionServerTracker-0] regionserver.HRegionServer(2295): ***** STOPPING region server 'jenkins-hbase20.apache.org,39195,1684727774188' ***** 2023-05-22 03:57:16,140 INFO [RegionServerTracker-0] regionserver.HRegionServer(2309): STOPPED: Cluster shutdown set; onlineServer=0 2023-05-22 03:57:16,141 DEBUG [M:0;jenkins-hbase20:39195] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b3a38c4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-05-22 03:57:16,141 INFO [M:0;jenkins-hbase20:39195] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,39195,1684727774188 2023-05-22 03:57:16,141 INFO [M:0;jenkins-hbase20:39195] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,39195,1684727774188; all regions closed. 2023-05-22 03:57:16,141 DEBUG [M:0;jenkins-hbase20:39195] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:57:16,141 DEBUG [M:0;jenkins-hbase20:39195] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-05-22 03:57:16,141 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-05-22 03:57:16,141 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727774365] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727774365,5,FailOnTimeoutGroup] 2023-05-22 03:57:16,141 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727774365] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727774365,5,FailOnTimeoutGroup] 2023-05-22 03:57:16,141 DEBUG [M:0;jenkins-hbase20:39195] cleaner.HFileCleaner(317): Stopping file delete threads 2023-05-22 03:57:16,143 INFO [M:0;jenkins-hbase20:39195] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-05-22 03:57:16,143 INFO [M:0;jenkins-hbase20:39195] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-05-22 03:57:16,143 INFO [M:0;jenkins-hbase20:39195] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-05-22 03:57:16,143 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-05-22 03:57:16,143 DEBUG [M:0;jenkins-hbase20:39195] master.HMaster(1512): Stopping service threads 2023-05-22 03:57:16,143 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:57:16,143 INFO [M:0;jenkins-hbase20:39195] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-05-22 03:57:16,143 ERROR [M:0;jenkins-hbase20:39195] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-05-22 03:57:16,144 INFO [M:0;jenkins-hbase20:39195] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-05-22 03:57:16,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:57:16,144 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-05-22 03:57:16,144 DEBUG [M:0;jenkins-hbase20:39195] zookeeper.ZKUtil(398): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-05-22 03:57:16,144 WARN [M:0;jenkins-hbase20:39195] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-05-22 03:57:16,144 INFO [M:0;jenkins-hbase20:39195] assignment.AssignmentManager(315): Stopping assignment manager 2023-05-22 03:57:16,145 INFO [M:0;jenkins-hbase20:39195] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-05-22 03:57:16,145 DEBUG [M:0;jenkins-hbase20:39195] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-05-22 03:57:16,145 INFO [M:0;jenkins-hbase20:39195] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:57:16,145 DEBUG [M:0;jenkins-hbase20:39195] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:57:16,145 DEBUG [M:0;jenkins-hbase20:39195] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-05-22 03:57:16,145 DEBUG [M:0;jenkins-hbase20:39195] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:57:16,145 INFO [M:0;jenkins-hbase20:39195] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=38.94 KB heapSize=47.38 KB 2023-05-22 03:57:16,155 INFO [M:0;jenkins-hbase20:39195] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.94 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e8cb2043d94a4a7290aa667add69a103 2023-05-22 03:57:16,159 INFO [M:0;jenkins-hbase20:39195] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e8cb2043d94a4a7290aa667add69a103 2023-05-22 03:57:16,160 DEBUG [M:0;jenkins-hbase20:39195] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e8cb2043d94a4a7290aa667add69a103 as hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e8cb2043d94a4a7290aa667add69a103 2023-05-22 03:57:16,164 INFO [M:0;jenkins-hbase20:39195] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e8cb2043d94a4a7290aa667add69a103 2023-05-22 03:57:16,164 INFO [M:0;jenkins-hbase20:39195] regionserver.HStore(1080): Added hdfs://localhost.localdomain:42989/user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e8cb2043d94a4a7290aa667add69a103, entries=11, sequenceid=100, filesize=6.1 K 2023-05-22 03:57:16,165 INFO [M:0;jenkins-hbase20:39195] regionserver.HRegion(2948): Finished flush of dataSize ~38.94 KB/39878, heapSize ~47.36 KB/48496, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 20ms, sequenceid=100, compaction requested=false 2023-05-22 03:57:16,166 INFO [M:0;jenkins-hbase20:39195] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:57:16,167 DEBUG [M:0;jenkins-hbase20:39195] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:57:16,167 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/caafea4f-6d6e-ed0d-ea87-4fecbc36f67e/MasterData/WALs/jenkins-hbase20.apache.org,39195,1684727774188 2023-05-22 03:57:16,170 INFO [M:0;jenkins-hbase20:39195] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-05-22 03:57:16,170 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-05-22 03:57:16,170 INFO [M:0;jenkins-hbase20:39195] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:39195 2023-05-22 03:57:16,171 DEBUG [M:0;jenkins-hbase20:39195] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,39195,1684727774188 already deleted, retry=false 2023-05-22 03:57:16,239 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:57:16,239 INFO [RS:0;jenkins-hbase20:40647] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,40647,1684727774227; zookeeper connection closed. 2023-05-22 03:57:16,239 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): regionserver:40647-0x101714fd4060001, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:57:16,239 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@26f05d1a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@26f05d1a 2023-05-22 03:57:16,240 INFO [Listener at localhost.localdomain/35753] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-05-22 03:57:16,339 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:57:16,340 DEBUG [Listener at localhost.localdomain/35753-EventThread] zookeeper.ZKWatcher(600): master:39195-0x101714fd4060000, quorum=127.0.0.1:58947, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:57:16,339 INFO [M:0;jenkins-hbase20:39195] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,39195,1684727774188; zookeeper connection closed. 2023-05-22 03:57:16,341 WARN [Listener at localhost.localdomain/35753] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:57:16,347 INFO [Listener at localhost.localdomain/35753] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:57:16,452 WARN [BP-639809515-148.251.75.209-1684727773740 heartbeating to localhost.localdomain/127.0.0.1:42989] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:57:16,452 WARN [BP-639809515-148.251.75.209-1684727773740 heartbeating to localhost.localdomain/127.0.0.1:42989] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-639809515-148.251.75.209-1684727773740 (Datanode Uuid 41a3d320-5358-4527-be98-ce2d395bea19) service to localhost.localdomain/127.0.0.1:42989 2023-05-22 03:57:16,453 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/cluster_d6f4cf05-5962-2ada-e7eb-e39cba3bab08/dfs/data/data3/current/BP-639809515-148.251.75.209-1684727773740] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:57:16,454 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/cluster_d6f4cf05-5962-2ada-e7eb-e39cba3bab08/dfs/data/data4/current/BP-639809515-148.251.75.209-1684727773740] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:57:16,455 WARN [Listener at localhost.localdomain/35753] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:57:16,459 INFO [Listener at localhost.localdomain/35753] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:57:16,498 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-05-22 03:57:16,565 WARN [BP-639809515-148.251.75.209-1684727773740 heartbeating to localhost.localdomain/127.0.0.1:42989] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:57:16,565 WARN [BP-639809515-148.251.75.209-1684727773740 heartbeating to localhost.localdomain/127.0.0.1:42989] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-639809515-148.251.75.209-1684727773740 (Datanode Uuid 146c2160-e757-4007-9eaa-d82aaee88864) service to localhost.localdomain/127.0.0.1:42989 2023-05-22 03:57:16,567 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/cluster_d6f4cf05-5962-2ada-e7eb-e39cba3bab08/dfs/data/data1/current/BP-639809515-148.251.75.209-1684727773740] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:57:16,567 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/cluster_d6f4cf05-5962-2ada-e7eb-e39cba3bab08/dfs/data/data2/current/BP-639809515-148.251.75.209-1684727773740] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:57:16,583 INFO [Listener at localhost.localdomain/35753] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-05-22 03:57:16,700 INFO [Listener at localhost.localdomain/35753] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-05-22 03:57:16,716 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-05-22 03:57:16,724 INFO [Listener at localhost.localdomain/35753] hbase.ResourceChecker(175): after: regionserver.wal.TestLogRolling#testCompactionRecordDoesntBlockRolling Thread=95 (was 88) - Thread LEAK? -, OpenFileDescriptor=499 (was 461) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=53 (was 110), ProcessCount=165 (was 175), AvailableMemoryMB=6601 (was 6909) 2023-05-22 03:57:16,731 INFO [Listener at localhost.localdomain/35753] hbase.ResourceChecker(147): before: regionserver.wal.TestLogRolling#testLogRolling Thread=96, OpenFileDescriptor=499, MaxFileDescriptor=60000, SystemLoadAverage=53, ProcessCount=165, AvailableMemoryMB=6600 2023-05-22 03:57:16,731 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-05-22 03:57:16,731 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/hadoop.log.dir so I do NOT create it in target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8 2023-05-22 03:57:16,731 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dec23939-b8a6-5e9d-fcf0-86329deae167/hadoop.tmp.dir so I do NOT create it in target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8 2023-05-22 03:57:16,732 INFO [Listener at localhost.localdomain/35753] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/cluster_13fb7289-2763-9201-0a7f-ce2633f64fd7, deleteOnExit=true 2023-05-22 03:57:16,732 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-05-22 03:57:16,732 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/test.cache.data in system properties and HBase conf 2023-05-22 03:57:16,732 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/hadoop.tmp.dir in system properties and HBase conf 2023-05-22 03:57:16,732 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/hadoop.log.dir in system properties and HBase conf 2023-05-22 03:57:16,732 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/mapreduce.cluster.local.dir in system properties and HBase conf 2023-05-22 03:57:16,732 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-05-22 03:57:16,732 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-05-22 03:57:16,733 DEBUG [Listener at localhost.localdomain/35753] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-05-22 03:57:16,733 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-05-22 03:57:16,733 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-05-22 03:57:16,733 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-05-22 03:57:16,733 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-05-22 03:57:16,733 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-05-22 03:57:16,733 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-05-22 03:57:16,734 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-05-22 03:57:16,734 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/dfs.journalnode.edits.dir in system properties and HBase conf 2023-05-22 03:57:16,734 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-05-22 03:57:16,734 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/nfs.dump.dir in system properties and HBase conf 2023-05-22 03:57:16,734 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/java.io.tmpdir in system properties and HBase conf 2023-05-22 03:57:16,734 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/dfs.journalnode.edits.dir in system properties and HBase conf 2023-05-22 03:57:16,734 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-05-22 03:57:16,734 INFO [Listener at localhost.localdomain/35753] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-05-22 03:57:16,736 WARN [Listener at localhost.localdomain/35753] blockmanagement.DatanodeManager(362): The given interval for marking stale datanode = 30000, which is larger than heartbeat expire interval 20000. 2023-05-22 03:57:16,737 WARN [Listener at localhost.localdomain/35753] conf.Configuration(1701): No unit for dfs.heartbeat.interval(1) assuming SECONDS 2023-05-22 03:57:16,737 WARN [Listener at localhost.localdomain/35753] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-05-22 03:57:16,760 WARN [Listener at localhost.localdomain/35753] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:57:16,763 INFO [Listener at localhost.localdomain/35753] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:57:16,769 INFO [Listener at localhost.localdomain/35753] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/java.io.tmpdir/Jetty_localhost_localdomain_43797_hdfs____w82j68/webapp 2023-05-22 03:57:16,841 INFO [Listener at localhost.localdomain/35753] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:43797 2023-05-22 03:57:16,843 WARN [Listener at localhost.localdomain/35753] blockmanagement.DatanodeManager(362): The given interval for marking stale datanode = 30000, which is larger than heartbeat expire interval 20000. 2023-05-22 03:57:16,844 WARN [Listener at localhost.localdomain/35753] conf.Configuration(1701): No unit for dfs.heartbeat.interval(1) assuming SECONDS 2023-05-22 03:57:16,844 WARN [Listener at localhost.localdomain/35753] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-05-22 03:57:16,867 WARN [Listener at localhost.localdomain/45435] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:57:16,876 WARN [Listener at localhost.localdomain/45435] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:57:16,878 WARN [Listener at localhost.localdomain/45435] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:57:16,880 INFO [Listener at localhost.localdomain/45435] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:57:16,886 INFO [Listener at localhost.localdomain/45435] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/java.io.tmpdir/Jetty_localhost_37009_datanode____u1ni64/webapp 2023-05-22 03:57:16,960 INFO [Listener at localhost.localdomain/45435] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37009 2023-05-22 03:57:16,965 WARN [Listener at localhost.localdomain/44521] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:57:16,974 WARN [Listener at localhost.localdomain/44521] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:57:16,977 WARN [Listener at localhost.localdomain/44521] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:57:16,978 INFO [Listener at localhost.localdomain/44521] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:57:16,980 INFO [Listener at localhost.localdomain/44521] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/java.io.tmpdir/Jetty_localhost_45685_datanode____td6j5l/webapp 2023-05-22 03:57:17,035 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xba6bd26b226ba4c3: Processing first storage report for DS-2d15a1ed-d313-4888-a97e-9e66f2a56afd from datanode 39c20281-e2ec-4825-9b9d-2c5aee149ecf 2023-05-22 03:57:17,035 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xba6bd26b226ba4c3: from storage DS-2d15a1ed-d313-4888-a97e-9e66f2a56afd node DatanodeRegistration(127.0.0.1:46555, datanodeUuid=39c20281-e2ec-4825-9b9d-2c5aee149ecf, infoPort=41221, infoSecurePort=0, ipcPort=44521, storageInfo=lv=-57;cid=testClusterID;nsid=1838601561;c=1684727836738), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-05-22 03:57:17,035 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xba6bd26b226ba4c3: Processing first storage report for DS-6938ee6e-9617-4adf-9576-f34799f04571 from datanode 39c20281-e2ec-4825-9b9d-2c5aee149ecf 2023-05-22 03:57:17,035 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xba6bd26b226ba4c3: from storage DS-6938ee6e-9617-4adf-9576-f34799f04571 node DatanodeRegistration(127.0.0.1:46555, datanodeUuid=39c20281-e2ec-4825-9b9d-2c5aee149ecf, infoPort=41221, infoSecurePort=0, ipcPort=44521, storageInfo=lv=-57;cid=testClusterID;nsid=1838601561;c=1684727836738), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:57:17,067 INFO [Listener at localhost.localdomain/44521] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45685 2023-05-22 03:57:17,073 WARN [Listener at localhost.localdomain/33915] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:57:17,124 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x95089c68d5660518: Processing first storage report for DS-c8310352-76ac-432e-a8a9-39ce69f5cab1 from datanode e53251e3-b749-4350-8e50-b4b41a9a4116 2023-05-22 03:57:17,124 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x95089c68d5660518: from storage DS-c8310352-76ac-432e-a8a9-39ce69f5cab1 node DatanodeRegistration(127.0.0.1:39185, datanodeUuid=e53251e3-b749-4350-8e50-b4b41a9a4116, infoPort=35683, infoSecurePort=0, ipcPort=33915, storageInfo=lv=-57;cid=testClusterID;nsid=1838601561;c=1684727836738), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:57:17,124 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x95089c68d5660518: Processing first storage report for DS-e75196ab-7cf8-47c7-bfc4-c7a4a3540e0a from datanode e53251e3-b749-4350-8e50-b4b41a9a4116 2023-05-22 03:57:17,124 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x95089c68d5660518: from storage DS-e75196ab-7cf8-47c7-bfc4-c7a4a3540e0a node DatanodeRegistration(127.0.0.1:39185, datanodeUuid=e53251e3-b749-4350-8e50-b4b41a9a4116, infoPort=35683, infoSecurePort=0, ipcPort=33915, storageInfo=lv=-57;cid=testClusterID;nsid=1838601561;c=1684727836738), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:57:17,183 DEBUG [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8 2023-05-22 03:57:17,186 INFO [Listener at localhost.localdomain/33915] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/cluster_13fb7289-2763-9201-0a7f-ce2633f64fd7/zookeeper_0, clientPort=59657, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/cluster_13fb7289-2763-9201-0a7f-ce2633f64fd7/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/cluster_13fb7289-2763-9201-0a7f-ce2633f64fd7/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-05-22 03:57:17,188 INFO [Listener at localhost.localdomain/33915] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=59657 2023-05-22 03:57:17,189 INFO [Listener at localhost.localdomain/33915] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:57:17,191 INFO [Listener at localhost.localdomain/33915] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:57:17,207 INFO [Listener at localhost.localdomain/33915] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24 with version=8 2023-05-22 03:57:17,207 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/hbase-staging 2023-05-22 03:57:17,209 INFO [Listener at localhost.localdomain/33915] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-05-22 03:57:17,209 INFO [Listener at localhost.localdomain/33915] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:57:17,209 INFO [Listener at localhost.localdomain/33915] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-05-22 03:57:17,209 INFO [Listener at localhost.localdomain/33915] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-05-22 03:57:17,209 INFO [Listener at localhost.localdomain/33915] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:57:17,209 INFO [Listener at localhost.localdomain/33915] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-05-22 03:57:17,209 INFO [Listener at localhost.localdomain/33915] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-05-22 03:57:17,211 INFO [Listener at localhost.localdomain/33915] ipc.NettyRpcServer(120): Bind to /148.251.75.209:43369 2023-05-22 03:57:17,211 INFO [Listener at localhost.localdomain/33915] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:57:17,212 INFO [Listener at localhost.localdomain/33915] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:57:17,213 INFO [Listener at localhost.localdomain/33915] zookeeper.RecoverableZooKeeper(93): Process identifier=master:43369 connecting to ZooKeeper ensemble=127.0.0.1:59657 2023-05-22 03:57:17,217 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:433690x0, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-05-22 03:57:17,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:43369-0x1017150ca2d0000 connected 2023-05-22 03:57:17,268 DEBUG [Listener at localhost.localdomain/33915] zookeeper.ZKUtil(164): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:57:17,268 DEBUG [Listener at localhost.localdomain/33915] zookeeper.ZKUtil(164): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:57:17,269 DEBUG [Listener at localhost.localdomain/33915] zookeeper.ZKUtil(164): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-05-22 03:57:17,269 DEBUG [Listener at localhost.localdomain/33915] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43369 2023-05-22 03:57:17,269 DEBUG [Listener at localhost.localdomain/33915] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43369 2023-05-22 03:57:17,270 DEBUG [Listener at localhost.localdomain/33915] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43369 2023-05-22 03:57:17,270 DEBUG [Listener at localhost.localdomain/33915] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43369 2023-05-22 03:57:17,275 DEBUG [Listener at localhost.localdomain/33915] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43369 2023-05-22 03:57:17,275 INFO [Listener at localhost.localdomain/33915] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24, hbase.cluster.distributed=false 2023-05-22 03:57:17,286 INFO [Listener at localhost.localdomain/33915] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-05-22 03:57:17,286 INFO [Listener at localhost.localdomain/33915] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:57:17,286 INFO [Listener at localhost.localdomain/33915] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-05-22 03:57:17,286 INFO [Listener at localhost.localdomain/33915] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-05-22 03:57:17,286 INFO [Listener at localhost.localdomain/33915] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:57:17,287 INFO [Listener at localhost.localdomain/33915] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-05-22 03:57:17,287 INFO [Listener at localhost.localdomain/33915] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-05-22 03:57:17,288 INFO [Listener at localhost.localdomain/33915] ipc.NettyRpcServer(120): Bind to /148.251.75.209:43747 2023-05-22 03:57:17,288 INFO [Listener at localhost.localdomain/33915] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-05-22 03:57:17,289 DEBUG [Listener at localhost.localdomain/33915] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-05-22 03:57:17,290 INFO [Listener at localhost.localdomain/33915] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:57:17,291 INFO [Listener at localhost.localdomain/33915] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:57:17,291 INFO [Listener at localhost.localdomain/33915] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43747 connecting to ZooKeeper ensemble=127.0.0.1:59657 2023-05-22 03:57:17,294 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): regionserver:437470x0, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-05-22 03:57:17,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:43747-0x1017150ca2d0001 connected 2023-05-22 03:57:17,296 DEBUG [Listener at localhost.localdomain/33915] zookeeper.ZKUtil(164): regionserver:43747-0x1017150ca2d0001, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:57:17,296 DEBUG [Listener at localhost.localdomain/33915] zookeeper.ZKUtil(164): regionserver:43747-0x1017150ca2d0001, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:57:17,297 DEBUG [Listener at localhost.localdomain/33915] zookeeper.ZKUtil(164): regionserver:43747-0x1017150ca2d0001, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-05-22 03:57:17,297 DEBUG [Listener at localhost.localdomain/33915] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43747 2023-05-22 03:57:17,297 DEBUG [Listener at localhost.localdomain/33915] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43747 2023-05-22 03:57:17,297 DEBUG [Listener at localhost.localdomain/33915] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43747 2023-05-22 03:57:17,298 DEBUG [Listener at localhost.localdomain/33915] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43747 2023-05-22 03:57:17,298 DEBUG [Listener at localhost.localdomain/33915] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43747 2023-05-22 03:57:17,299 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,43369,1684727837208 2023-05-22 03:57:17,300 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-05-22 03:57:17,301 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,43369,1684727837208 2023-05-22 03:57:17,301 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-05-22 03:57:17,301 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): regionserver:43747-0x1017150ca2d0001, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-05-22 03:57:17,301 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:57:17,302 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-05-22 03:57:17,303 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,43369,1684727837208 from backup master directory 2023-05-22 03:57:17,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-05-22 03:57:17,304 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,43369,1684727837208 2023-05-22 03:57:17,304 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-05-22 03:57:17,304 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-05-22 03:57:17,304 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,43369,1684727837208 2023-05-22 03:57:17,316 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/hbase.id with ID: 4bf956be-693a-476d-aefb-0626b44713b9 2023-05-22 03:57:17,329 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:57:17,331 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:57:17,338 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x579aa482 to 127.0.0.1:59657 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:57:17,344 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54a49c7e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:57:17,345 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-05-22 03:57:17,345 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-05-22 03:57:17,345 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:57:17,347 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/MasterData/data/master/store-tmp 2023-05-22 03:57:17,355 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:57:17,355 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-05-22 03:57:17,355 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:57:17,355 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:57:17,355 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-05-22 03:57:17,355 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:57:17,355 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:57:17,355 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:57:17,356 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/MasterData/WALs/jenkins-hbase20.apache.org,43369,1684727837208 2023-05-22 03:57:17,359 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C43369%2C1684727837208, suffix=, logDir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/MasterData/WALs/jenkins-hbase20.apache.org,43369,1684727837208, archiveDir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/MasterData/oldWALs, maxLogs=10 2023-05-22 03:57:17,366 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/MasterData/WALs/jenkins-hbase20.apache.org,43369,1684727837208/jenkins-hbase20.apache.org%2C43369%2C1684727837208.1684727837359 2023-05-22 03:57:17,366 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46555,DS-2d15a1ed-d313-4888-a97e-9e66f2a56afd,DISK], DatanodeInfoWithStorage[127.0.0.1:39185,DS-c8310352-76ac-432e-a8a9-39ce69f5cab1,DISK]] 2023-05-22 03:57:17,366 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:57:17,366 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:57:17,366 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:57:17,366 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:57:17,368 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:57:17,370 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-05-22 03:57:17,370 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-05-22 03:57:17,370 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:57:17,371 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:57:17,372 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:57:17,374 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:57:17,379 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:57:17,379 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=878112, jitterRate=0.11657790839672089}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:57:17,379 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:57:17,380 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-05-22 03:57:17,381 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-05-22 03:57:17,381 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-05-22 03:57:17,381 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-05-22 03:57:17,381 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-05-22 03:57:17,382 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-05-22 03:57:17,382 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-05-22 03:57:17,382 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-05-22 03:57:17,383 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-05-22 03:57:17,392 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-05-22 03:57:17,393 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-22 03:57:17,393 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-05-22 03:57:17,393 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-05-22 03:57:17,394 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-05-22 03:57:17,395 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:57:17,395 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-05-22 03:57:17,396 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-05-22 03:57:17,396 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-05-22 03:57:17,397 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-05-22 03:57:17,397 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): regionserver:43747-0x1017150ca2d0001, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-05-22 03:57:17,397 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:57:17,397 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,43369,1684727837208, sessionid=0x1017150ca2d0000, setting cluster-up flag (Was=false) 2023-05-22 03:57:17,400 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:57:17,402 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-05-22 03:57:17,402 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,43369,1684727837208 2023-05-22 03:57:17,404 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:57:17,406 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-05-22 03:57:17,407 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,43369,1684727837208 2023-05-22 03:57:17,407 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/.hbase-snapshot/.tmp 2023-05-22 03:57:17,411 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-05-22 03:57:17,412 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:57:17,412 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:57:17,412 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:57:17,412 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:57:17,412 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-05-22 03:57:17,412 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:57:17,412 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-05-22 03:57:17,412 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:57:17,413 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1684727867413 2023-05-22 03:57:17,413 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-05-22 03:57:17,413 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-05-22 03:57:17,413 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-05-22 03:57:17,413 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-05-22 03:57:17,414 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-05-22 03:57:17,414 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-05-22 03:57:17,414 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-05-22 03:57:17,414 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-05-22 03:57:17,414 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-05-22 03:57:17,414 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-05-22 03:57:17,414 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-05-22 03:57:17,414 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-05-22 03:57:17,415 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-05-22 03:57:17,415 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-05-22 03:57:17,415 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727837415,5,FailOnTimeoutGroup] 2023-05-22 03:57:17,415 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727837415,5,FailOnTimeoutGroup] 2023-05-22 03:57:17,415 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-05-22 03:57:17,415 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-05-22 03:57:17,415 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-05-22 03:57:17,415 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-05-22 03:57:17,415 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-05-22 03:57:17,426 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-05-22 03:57:17,427 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-05-22 03:57:17,427 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24 2023-05-22 03:57:17,436 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:57:17,437 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-05-22 03:57:17,438 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/info 2023-05-22 03:57:17,438 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-05-22 03:57:17,438 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:57:17,439 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-05-22 03:57:17,440 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:57:17,440 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-05-22 03:57:17,440 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:57:17,441 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-05-22 03:57:17,442 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/table 2023-05-22 03:57:17,442 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-05-22 03:57:17,442 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:57:17,443 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740 2023-05-22 03:57:17,443 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740 2023-05-22 03:57:17,445 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-05-22 03:57:17,445 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-05-22 03:57:17,447 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:57:17,447 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=728702, jitterRate=-0.07340797781944275}}}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-05-22 03:57:17,448 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-05-22 03:57:17,448 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-05-22 03:57:17,448 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-05-22 03:57:17,448 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-05-22 03:57:17,448 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-05-22 03:57:17,448 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-05-22 03:57:17,448 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-05-22 03:57:17,449 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-05-22 03:57:17,449 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-05-22 03:57:17,449 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-05-22 03:57:17,450 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-05-22 03:57:17,451 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-05-22 03:57:17,452 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-05-22 03:57:17,501 INFO [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(951): ClusterId : 4bf956be-693a-476d-aefb-0626b44713b9 2023-05-22 03:57:17,502 DEBUG [RS:0;jenkins-hbase20:43747] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-05-22 03:57:17,505 DEBUG [RS:0;jenkins-hbase20:43747] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-05-22 03:57:17,505 DEBUG [RS:0;jenkins-hbase20:43747] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-05-22 03:57:17,507 DEBUG [RS:0;jenkins-hbase20:43747] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-05-22 03:57:17,508 DEBUG [RS:0;jenkins-hbase20:43747] zookeeper.ReadOnlyZKClient(139): Connect 0x13a975ce to 127.0.0.1:59657 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:57:17,512 DEBUG [RS:0;jenkins-hbase20:43747] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d6e99cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:57:17,513 DEBUG [RS:0;jenkins-hbase20:43747] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19511ebe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-05-22 03:57:17,524 DEBUG [RS:0;jenkins-hbase20:43747] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:43747 2023-05-22 03:57:17,524 INFO [RS:0;jenkins-hbase20:43747] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-05-22 03:57:17,524 INFO [RS:0;jenkins-hbase20:43747] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-05-22 03:57:17,524 DEBUG [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(1022): About to register with Master. 2023-05-22 03:57:17,525 INFO [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(2809): reportForDuty to master=jenkins-hbase20.apache.org,43369,1684727837208 with isa=jenkins-hbase20.apache.org/148.251.75.209:43747, startcode=1684727837286 2023-05-22 03:57:17,525 DEBUG [RS:0;jenkins-hbase20:43747] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-05-22 03:57:17,528 INFO [RS-EventLoopGroup-12-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:56899, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.5 (auth:SIMPLE), service=RegionServerStatusService 2023-05-22 03:57:17,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43369] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:17,529 DEBUG [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24 2023-05-22 03:57:17,529 DEBUG [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:45435 2023-05-22 03:57:17,529 DEBUG [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-05-22 03:57:17,531 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:57:17,531 DEBUG [RS:0;jenkins-hbase20:43747] zookeeper.ZKUtil(162): regionserver:43747-0x1017150ca2d0001, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:17,531 WARN [RS:0;jenkins-hbase20:43747] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-05-22 03:57:17,531 INFO [RS:0;jenkins-hbase20:43747] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:57:17,531 DEBUG [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(1946): logDir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/WALs/jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:17,532 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,43747,1684727837286] 2023-05-22 03:57:17,535 DEBUG [RS:0;jenkins-hbase20:43747] zookeeper.ZKUtil(162): regionserver:43747-0x1017150ca2d0001, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:17,536 DEBUG [RS:0;jenkins-hbase20:43747] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-05-22 03:57:17,536 INFO [RS:0;jenkins-hbase20:43747] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-05-22 03:57:17,537 INFO [RS:0;jenkins-hbase20:43747] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-05-22 03:57:17,538 INFO [RS:0;jenkins-hbase20:43747] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-05-22 03:57:17,538 INFO [RS:0;jenkins-hbase20:43747] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:57:17,538 INFO [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-05-22 03:57:17,539 INFO [RS:0;jenkins-hbase20:43747] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-05-22 03:57:17,539 DEBUG [RS:0;jenkins-hbase20:43747] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:57:17,539 DEBUG [RS:0;jenkins-hbase20:43747] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:57:17,539 DEBUG [RS:0;jenkins-hbase20:43747] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:57:17,539 DEBUG [RS:0;jenkins-hbase20:43747] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:57:17,539 DEBUG [RS:0;jenkins-hbase20:43747] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:57:17,539 DEBUG [RS:0;jenkins-hbase20:43747] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-05-22 03:57:17,540 DEBUG [RS:0;jenkins-hbase20:43747] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:57:17,540 DEBUG [RS:0;jenkins-hbase20:43747] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:57:17,540 DEBUG [RS:0;jenkins-hbase20:43747] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:57:17,540 DEBUG [RS:0;jenkins-hbase20:43747] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:57:17,540 INFO [RS:0;jenkins-hbase20:43747] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-05-22 03:57:17,540 INFO [RS:0;jenkins-hbase20:43747] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-05-22 03:57:17,540 INFO [RS:0;jenkins-hbase20:43747] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-05-22 03:57:17,551 INFO [RS:0;jenkins-hbase20:43747] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-05-22 03:57:17,552 INFO [RS:0;jenkins-hbase20:43747] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43747,1684727837286-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:57:17,560 INFO [RS:0;jenkins-hbase20:43747] regionserver.Replication(203): jenkins-hbase20.apache.org,43747,1684727837286 started 2023-05-22 03:57:17,560 INFO [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,43747,1684727837286, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:43747, sessionid=0x1017150ca2d0001 2023-05-22 03:57:17,560 DEBUG [RS:0;jenkins-hbase20:43747] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-05-22 03:57:17,560 DEBUG [RS:0;jenkins-hbase20:43747] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:17,560 DEBUG [RS:0;jenkins-hbase20:43747] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,43747,1684727837286' 2023-05-22 03:57:17,560 DEBUG [RS:0;jenkins-hbase20:43747] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:57:17,561 DEBUG [RS:0;jenkins-hbase20:43747] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:57:17,561 DEBUG [RS:0;jenkins-hbase20:43747] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-05-22 03:57:17,561 DEBUG [RS:0;jenkins-hbase20:43747] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-05-22 03:57:17,561 DEBUG [RS:0;jenkins-hbase20:43747] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:17,561 DEBUG [RS:0;jenkins-hbase20:43747] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,43747,1684727837286' 2023-05-22 03:57:17,561 DEBUG [RS:0;jenkins-hbase20:43747] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-05-22 03:57:17,562 DEBUG [RS:0;jenkins-hbase20:43747] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-05-22 03:57:17,562 DEBUG [RS:0;jenkins-hbase20:43747] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-05-22 03:57:17,562 INFO [RS:0;jenkins-hbase20:43747] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-05-22 03:57:17,562 INFO [RS:0;jenkins-hbase20:43747] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-05-22 03:57:17,602 DEBUG [jenkins-hbase20:43369] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-05-22 03:57:17,603 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,43747,1684727837286, state=OPENING 2023-05-22 03:57:17,604 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-05-22 03:57:17,606 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:57:17,606 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,43747,1684727837286}] 2023-05-22 03:57:17,607 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-05-22 03:57:17,664 INFO [RS:0;jenkins-hbase20:43747] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C43747%2C1684727837286, suffix=, logDir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/WALs/jenkins-hbase20.apache.org,43747,1684727837286, archiveDir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/oldWALs, maxLogs=32 2023-05-22 03:57:17,674 INFO [RS:0;jenkins-hbase20:43747] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/WALs/jenkins-hbase20.apache.org,43747,1684727837286/jenkins-hbase20.apache.org%2C43747%2C1684727837286.1684727837664 2023-05-22 03:57:17,675 DEBUG [RS:0;jenkins-hbase20:43747] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46555,DS-2d15a1ed-d313-4888-a97e-9e66f2a56afd,DISK], DatanodeInfoWithStorage[127.0.0.1:39185,DS-c8310352-76ac-432e-a8a9-39ce69f5cab1,DISK]] 2023-05-22 03:57:17,764 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:17,764 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-05-22 03:57:17,772 INFO [RS-EventLoopGroup-13-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54926, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-05-22 03:57:17,778 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-05-22 03:57:17,779 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:57:17,781 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C43747%2C1684727837286.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/WALs/jenkins-hbase20.apache.org,43747,1684727837286, archiveDir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/oldWALs, maxLogs=32 2023-05-22 03:57:17,792 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/WALs/jenkins-hbase20.apache.org,43747,1684727837286/jenkins-hbase20.apache.org%2C43747%2C1684727837286.meta.1684727837781.meta 2023-05-22 03:57:17,792 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39185,DS-c8310352-76ac-432e-a8a9-39ce69f5cab1,DISK], DatanodeInfoWithStorage[127.0.0.1:46555,DS-2d15a1ed-d313-4888-a97e-9e66f2a56afd,DISK]] 2023-05-22 03:57:17,793 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:57:17,793 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-05-22 03:57:17,793 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-05-22 03:57:17,793 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-05-22 03:57:17,793 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-05-22 03:57:17,793 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:57:17,793 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-05-22 03:57:17,794 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-05-22 03:57:17,795 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-05-22 03:57:17,797 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/info 2023-05-22 03:57:17,797 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/info 2023-05-22 03:57:17,797 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-05-22 03:57:17,798 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:57:17,798 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-05-22 03:57:17,799 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:57:17,799 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:57:17,799 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-05-22 03:57:17,800 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:57:17,800 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-05-22 03:57:17,801 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/table 2023-05-22 03:57:17,801 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/table 2023-05-22 03:57:17,801 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-05-22 03:57:17,801 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:57:17,802 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740 2023-05-22 03:57:17,803 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740 2023-05-22 03:57:17,805 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-05-22 03:57:17,806 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-05-22 03:57:17,807 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=711062, jitterRate=-0.09583844244480133}}}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-05-22 03:57:17,807 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-05-22 03:57:17,809 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1684727837764 2023-05-22 03:57:17,812 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for hbase:meta,,1.1588230740 2023-05-22 03:57:17,812 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-05-22 03:57:17,813 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,43747,1684727837286, state=OPEN 2023-05-22 03:57:17,814 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-05-22 03:57:17,814 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-05-22 03:57:17,816 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-05-22 03:57:17,816 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,43747,1684727837286 in 208 msec 2023-05-22 03:57:17,818 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-05-22 03:57:17,818 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 367 msec 2023-05-22 03:57:17,819 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 408 msec 2023-05-22 03:57:17,819 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1684727837819, completionTime=-1 2023-05-22 03:57:17,820 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-05-22 03:57:17,820 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-05-22 03:57:17,823 DEBUG [hconnection-0x6d8dec42-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-05-22 03:57:17,825 INFO [RS-EventLoopGroup-13-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54940, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-05-22 03:57:17,826 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-05-22 03:57:17,826 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1684727897826 2023-05-22 03:57:17,826 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1684727957826 2023-05-22 03:57:17,826 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-05-22 03:57:17,831 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43369,1684727837208-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:57:17,831 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43369,1684727837208-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:57:17,832 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43369,1684727837208-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:57:17,832 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:43369, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:57:17,832 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-05-22 03:57:17,832 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-05-22 03:57:17,832 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-05-22 03:57:17,833 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-05-22 03:57:17,833 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-05-22 03:57:17,835 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-05-22 03:57:17,836 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-05-22 03:57:17,838 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/.tmp/data/hbase/namespace/89fa14d6dab8af32afc99240e7dd087f 2023-05-22 03:57:17,838 DEBUG [HFileArchiver-9] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/.tmp/data/hbase/namespace/89fa14d6dab8af32afc99240e7dd087f empty. 2023-05-22 03:57:17,838 DEBUG [HFileArchiver-9] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/.tmp/data/hbase/namespace/89fa14d6dab8af32afc99240e7dd087f 2023-05-22 03:57:17,839 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-05-22 03:57:17,849 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-05-22 03:57:17,850 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 89fa14d6dab8af32afc99240e7dd087f, NAME => 'hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/.tmp 2023-05-22 03:57:17,857 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:57:17,857 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 89fa14d6dab8af32afc99240e7dd087f, disabling compactions & flushes 2023-05-22 03:57:17,857 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f. 2023-05-22 03:57:17,857 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f. 2023-05-22 03:57:17,857 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f. after waiting 0 ms 2023-05-22 03:57:17,857 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f. 2023-05-22 03:57:17,858 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f. 2023-05-22 03:57:17,858 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 89fa14d6dab8af32afc99240e7dd087f: 2023-05-22 03:57:17,860 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-05-22 03:57:17,861 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727837860"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1684727837860"}]},"ts":"1684727837860"} 2023-05-22 03:57:17,863 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-05-22 03:57:17,863 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-05-22 03:57:17,864 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727837863"}]},"ts":"1684727837863"} 2023-05-22 03:57:17,865 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-05-22 03:57:17,869 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=89fa14d6dab8af32afc99240e7dd087f, ASSIGN}] 2023-05-22 03:57:17,871 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=89fa14d6dab8af32afc99240e7dd087f, ASSIGN 2023-05-22 03:57:17,872 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=89fa14d6dab8af32afc99240e7dd087f, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,43747,1684727837286; forceNewPlan=false, retain=false 2023-05-22 03:57:18,024 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=89fa14d6dab8af32afc99240e7dd087f, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:18,025 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727838024"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1684727838024"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1684727838024"}]},"ts":"1684727838024"} 2023-05-22 03:57:18,030 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 89fa14d6dab8af32afc99240e7dd087f, server=jenkins-hbase20.apache.org,43747,1684727837286}] 2023-05-22 03:57:18,193 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f. 2023-05-22 03:57:18,193 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 89fa14d6dab8af32afc99240e7dd087f, NAME => 'hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:57:18,193 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 89fa14d6dab8af32afc99240e7dd087f 2023-05-22 03:57:18,194 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:57:18,194 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 89fa14d6dab8af32afc99240e7dd087f 2023-05-22 03:57:18,194 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 89fa14d6dab8af32afc99240e7dd087f 2023-05-22 03:57:18,197 INFO [StoreOpener-89fa14d6dab8af32afc99240e7dd087f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 89fa14d6dab8af32afc99240e7dd087f 2023-05-22 03:57:18,199 DEBUG [StoreOpener-89fa14d6dab8af32afc99240e7dd087f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/namespace/89fa14d6dab8af32afc99240e7dd087f/info 2023-05-22 03:57:18,200 DEBUG [StoreOpener-89fa14d6dab8af32afc99240e7dd087f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/namespace/89fa14d6dab8af32afc99240e7dd087f/info 2023-05-22 03:57:18,200 INFO [StoreOpener-89fa14d6dab8af32afc99240e7dd087f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 89fa14d6dab8af32afc99240e7dd087f columnFamilyName info 2023-05-22 03:57:18,201 INFO [StoreOpener-89fa14d6dab8af32afc99240e7dd087f-1] regionserver.HStore(310): Store=89fa14d6dab8af32afc99240e7dd087f/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:57:18,203 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/namespace/89fa14d6dab8af32afc99240e7dd087f 2023-05-22 03:57:18,204 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/namespace/89fa14d6dab8af32afc99240e7dd087f 2023-05-22 03:57:18,207 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 89fa14d6dab8af32afc99240e7dd087f 2023-05-22 03:57:18,209 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/namespace/89fa14d6dab8af32afc99240e7dd087f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:57:18,210 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 89fa14d6dab8af32afc99240e7dd087f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=727098, jitterRate=-0.07544736564159393}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:57:18,210 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 89fa14d6dab8af32afc99240e7dd087f: 2023-05-22 03:57:18,212 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f., pid=6, masterSystemTime=1684727838184 2023-05-22 03:57:18,214 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f. 2023-05-22 03:57:18,214 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f. 2023-05-22 03:57:18,214 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=89fa14d6dab8af32afc99240e7dd087f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:18,214 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727838214"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1684727838214"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1684727838214"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1684727838214"}]},"ts":"1684727838214"} 2023-05-22 03:57:18,218 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-05-22 03:57:18,218 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 89fa14d6dab8af32afc99240e7dd087f, server=jenkins-hbase20.apache.org,43747,1684727837286 in 186 msec 2023-05-22 03:57:18,220 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-05-22 03:57:18,220 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=89fa14d6dab8af32afc99240e7dd087f, ASSIGN in 349 msec 2023-05-22 03:57:18,221 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-05-22 03:57:18,221 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727838221"}]},"ts":"1684727838221"} 2023-05-22 03:57:18,223 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-05-22 03:57:18,225 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-05-22 03:57:18,226 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 393 msec 2023-05-22 03:57:18,238 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-05-22 03:57:18,244 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:57:18,244 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:57:18,249 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-05-22 03:57:18,259 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:57:18,262 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 13 msec 2023-05-22 03:57:18,271 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-05-22 03:57:18,278 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:57:18,282 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 11 msec 2023-05-22 03:57:18,294 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-05-22 03:57:18,296 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-05-22 03:57:18,296 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 0.992sec 2023-05-22 03:57:18,296 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-05-22 03:57:18,296 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-05-22 03:57:18,296 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-05-22 03:57:18,296 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43369,1684727837208-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-05-22 03:57:18,296 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43369,1684727837208-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-05-22 03:57:18,298 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-05-22 03:57:18,302 DEBUG [Listener at localhost.localdomain/33915] zookeeper.ReadOnlyZKClient(139): Connect 0x78d0f774 to 127.0.0.1:59657 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:57:18,311 DEBUG [Listener at localhost.localdomain/33915] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ddf9d80, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:57:18,313 DEBUG [hconnection-0x6bff79ce-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-05-22 03:57:18,314 INFO [RS-EventLoopGroup-13-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54942, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-05-22 03:57:18,316 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,43369,1684727837208 2023-05-22 03:57:18,316 INFO [Listener at localhost.localdomain/33915] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:57:18,320 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-05-22 03:57:18,320 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:57:18,320 INFO [Listener at localhost.localdomain/33915] master.MasterRpcServices(492): Client=null/null set balanceSwitch=false 2023-05-22 03:57:18,322 DEBUG [Listener at localhost.localdomain/33915] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-05-22 03:57:18,328 INFO [RS-EventLoopGroup-12-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55356, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-05-22 03:57:18,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43369] util.TableDescriptorChecker(340): MAX_FILESIZE for table descriptor or "hbase.hregion.max.filesize" (786432) is too small, which might cause over splitting into unmanageable number of regions. 2023-05-22 03:57:18,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43369] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (8192) is too small, which might cause very frequent flushing. 2023-05-22 03:57:18,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43369] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestLogRolling-testLogRolling', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-05-22 03:57:18,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43369] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestLogRolling-testLogRolling 2023-05-22 03:57:18,334 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestLogRolling-testLogRolling execute state=CREATE_TABLE_PRE_OPERATION 2023-05-22 03:57:18,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43369] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestLogRolling-testLogRolling" procId is: 9 2023-05-22 03:57:18,335 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestLogRolling-testLogRolling execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-05-22 03:57:18,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-05-22 03:57:18,337 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/.tmp/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:18,337 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/.tmp/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35 empty. 2023-05-22 03:57:18,338 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/.tmp/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:18,338 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestLogRolling-testLogRolling regions 2023-05-22 03:57:18,347 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/.tmp/data/default/TestLogRolling-testLogRolling/.tabledesc/.tableinfo.0000000001 2023-05-22 03:57:18,348 INFO [RegionOpenAndInit-TestLogRolling-testLogRolling-pool-0] regionserver.HRegion(7675): creating {ENCODED => 253e196376ee288bc24277fe737a9a35, NAME => 'TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestLogRolling-testLogRolling', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/.tmp 2023-05-22 03:57:18,356 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRolling-pool-0] regionserver.HRegion(866): Instantiated TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:57:18,356 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRolling-pool-0] regionserver.HRegion(1604): Closing 253e196376ee288bc24277fe737a9a35, disabling compactions & flushes 2023-05-22 03:57:18,356 INFO [RegionOpenAndInit-TestLogRolling-testLogRolling-pool-0] regionserver.HRegion(1626): Closing region TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. 2023-05-22 03:57:18,356 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRolling-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. 2023-05-22 03:57:18,356 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRolling-pool-0] regionserver.HRegion(1714): Acquired close lock on TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. after waiting 0 ms 2023-05-22 03:57:18,356 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRolling-pool-0] regionserver.HRegion(1724): Updates disabled for region TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. 2023-05-22 03:57:18,356 INFO [RegionOpenAndInit-TestLogRolling-testLogRolling-pool-0] regionserver.HRegion(1838): Closed TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. 2023-05-22 03:57:18,356 DEBUG [RegionOpenAndInit-TestLogRolling-testLogRolling-pool-0] regionserver.HRegion(1558): Region close journal for 253e196376ee288bc24277fe737a9a35: 2023-05-22 03:57:18,359 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestLogRolling-testLogRolling execute state=CREATE_TABLE_ADD_TO_META 2023-05-22 03:57:18,360 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1684727838359"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1684727838359"}]},"ts":"1684727838359"} 2023-05-22 03:57:18,361 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-05-22 03:57:18,362 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestLogRolling-testLogRolling execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-05-22 03:57:18,363 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestLogRolling-testLogRolling","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727838362"}]},"ts":"1684727838362"} 2023-05-22 03:57:18,364 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestLogRolling-testLogRolling, state=ENABLING in hbase:meta 2023-05-22 03:57:18,370 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestLogRolling-testLogRolling, region=253e196376ee288bc24277fe737a9a35, ASSIGN}] 2023-05-22 03:57:18,372 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestLogRolling-testLogRolling, region=253e196376ee288bc24277fe737a9a35, ASSIGN 2023-05-22 03:57:18,373 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestLogRolling-testLogRolling, region=253e196376ee288bc24277fe737a9a35, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,43747,1684727837286; forceNewPlan=false, retain=false 2023-05-22 03:57:18,524 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=253e196376ee288bc24277fe737a9a35, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:18,524 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1684727838524"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1684727838524"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1684727838524"}]},"ts":"1684727838524"} 2023-05-22 03:57:18,526 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 253e196376ee288bc24277fe737a9a35, server=jenkins-hbase20.apache.org,43747,1684727837286}] 2023-05-22 03:57:18,686 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. 2023-05-22 03:57:18,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 253e196376ee288bc24277fe737a9a35, NAME => 'TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:57:18,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestLogRolling-testLogRolling 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:18,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:57:18,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:18,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:18,691 INFO [StoreOpener-253e196376ee288bc24277fe737a9a35-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:18,693 DEBUG [StoreOpener-253e196376ee288bc24277fe737a9a35-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info 2023-05-22 03:57:18,694 DEBUG [StoreOpener-253e196376ee288bc24277fe737a9a35-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info 2023-05-22 03:57:18,694 INFO [StoreOpener-253e196376ee288bc24277fe737a9a35-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 253e196376ee288bc24277fe737a9a35 columnFamilyName info 2023-05-22 03:57:18,695 INFO [StoreOpener-253e196376ee288bc24277fe737a9a35-1] regionserver.HStore(310): Store=253e196376ee288bc24277fe737a9a35/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:57:18,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:18,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:18,699 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:18,702 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:57:18,702 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 253e196376ee288bc24277fe737a9a35; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=823416, jitterRate=0.047028109431266785}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:57:18,702 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 253e196376ee288bc24277fe737a9a35: 2023-05-22 03:57:18,703 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35., pid=11, masterSystemTime=1684727838679 2023-05-22 03:57:18,706 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. 2023-05-22 03:57:18,706 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. 2023-05-22 03:57:18,708 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=253e196376ee288bc24277fe737a9a35, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:18,708 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1684727838707"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1684727838707"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1684727838707"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1684727838707"}]},"ts":"1684727838707"} 2023-05-22 03:57:18,713 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-05-22 03:57:18,714 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 253e196376ee288bc24277fe737a9a35, server=jenkins-hbase20.apache.org,43747,1684727837286 in 184 msec 2023-05-22 03:57:18,716 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-05-22 03:57:18,716 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestLogRolling-testLogRolling, region=253e196376ee288bc24277fe737a9a35, ASSIGN in 344 msec 2023-05-22 03:57:18,717 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestLogRolling-testLogRolling execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-05-22 03:57:18,717 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestLogRolling-testLogRolling","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727838717"}]},"ts":"1684727838717"} 2023-05-22 03:57:18,718 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestLogRolling-testLogRolling, state=ENABLED in hbase:meta 2023-05-22 03:57:18,720 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestLogRolling-testLogRolling execute state=CREATE_TABLE_POST_OPERATION 2023-05-22 03:57:18,721 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestLogRolling-testLogRolling in 390 msec 2023-05-22 03:57:21,524 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-05-22 03:57:23,536 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-05-22 03:57:23,538 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-05-22 03:57:23,539 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestLogRolling-testLogRolling' 2023-05-22 03:57:28,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-05-22 03:57:28,339 INFO [Listener at localhost.localdomain/33915] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestLogRolling-testLogRolling, procId: 9 completed 2023-05-22 03:57:28,345 DEBUG [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(2627): Found 1 regions for table TestLogRolling-testLogRolling 2023-05-22 03:57:28,345 DEBUG [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(2633): firstRegionName=TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. 2023-05-22 03:57:28,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:28,364 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 253e196376ee288bc24277fe737a9a35 1/1 column families, dataSize=7.36 KB heapSize=8.13 KB 2023-05-22 03:57:28,378 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=7.36 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.tmp/info/dd9050d8078445bdb89e2358a367e7df 2023-05-22 03:57:28,389 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.tmp/info/dd9050d8078445bdb89e2358a367e7df as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/dd9050d8078445bdb89e2358a367e7df 2023-05-22 03:57:28,397 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/dd9050d8078445bdb89e2358a367e7df, entries=7, sequenceid=11, filesize=12.1 K 2023-05-22 03:57:28,398 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~7.36 KB/7532, heapSize ~8.11 KB/8304, currentSize=16.81 KB/17216 for 253e196376ee288bc24277fe737a9a35 in 34ms, sequenceid=11, compaction requested=false 2023-05-22 03:57:28,399 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 253e196376ee288bc24277fe737a9a35: 2023-05-22 03:57:28,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:28,400 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 253e196376ee288bc24277fe737a9a35 1/1 column families, dataSize=18.91 KB heapSize=20.50 KB 2023-05-22 03:57:28,420 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.91 KB at sequenceid=32 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.tmp/info/ea0838ae245c4f1081b46b2fccb50e3b 2023-05-22 03:57:28,427 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.tmp/info/ea0838ae245c4f1081b46b2fccb50e3b as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/ea0838ae245c4f1081b46b2fccb50e3b 2023-05-22 03:57:28,431 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/ea0838ae245c4f1081b46b2fccb50e3b, entries=18, sequenceid=32, filesize=23.7 K 2023-05-22 03:57:28,432 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~18.91 KB/19368, heapSize ~20.48 KB/20976, currentSize=7.36 KB/7532 for 253e196376ee288bc24277fe737a9a35 in 32ms, sequenceid=32, compaction requested=false 2023-05-22 03:57:28,432 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 253e196376ee288bc24277fe737a9a35: 2023-05-22 03:57:28,432 DEBUG [MemStoreFlusher.0] regionserver.ConstantSizeRegionSplitPolicy(109): Should split because info size=35.8 K, sizeToCheck=16.0 K 2023-05-22 03:57:28,432 DEBUG [MemStoreFlusher.0] regionserver.IncreasingToUpperBoundRegionSplitPolicy(84): regionsWithCommonTable=1 2023-05-22 03:57:28,432 DEBUG [MemStoreFlusher.0] regionserver.StoreUtils(129): cannot split hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/ea0838ae245c4f1081b46b2fccb50e3b because midkey is the same as first or last row 2023-05-22 03:57:30,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:30,423 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 253e196376ee288bc24277fe737a9a35 1/1 column families, dataSize=8.41 KB heapSize=9.25 KB 2023-05-22 03:57:30,441 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.41 KB at sequenceid=43 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.tmp/info/92ca70f0c39a4769b3277dbeef6fa788 2023-05-22 03:57:30,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.tmp/info/92ca70f0c39a4769b3277dbeef6fa788 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/92ca70f0c39a4769b3277dbeef6fa788 2023-05-22 03:57:30,454 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/92ca70f0c39a4769b3277dbeef6fa788, entries=8, sequenceid=43, filesize=13.2 K 2023-05-22 03:57:30,455 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~8.41 KB/8608, heapSize ~9.23 KB/9456, currentSize=16.81 KB/17216 for 253e196376ee288bc24277fe737a9a35 in 32ms, sequenceid=43, compaction requested=true 2023-05-22 03:57:30,455 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 253e196376ee288bc24277fe737a9a35: 2023-05-22 03:57:30,455 DEBUG [MemStoreFlusher.0] regionserver.ConstantSizeRegionSplitPolicy(109): Should split because info size=49.0 K, sizeToCheck=16.0 K 2023-05-22 03:57:30,455 DEBUG [MemStoreFlusher.0] regionserver.IncreasingToUpperBoundRegionSplitPolicy(84): regionsWithCommonTable=1 2023-05-22 03:57:30,455 DEBUG [MemStoreFlusher.0] regionserver.StoreUtils(129): cannot split hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/ea0838ae245c4f1081b46b2fccb50e3b because midkey is the same as first or last row 2023-05-22 03:57:30,456 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:57:30,456 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-05-22 03:57:30,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:30,457 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 253e196376ee288bc24277fe737a9a35 1/1 column families, dataSize=18.91 KB heapSize=20.50 KB 2023-05-22 03:57:30,458 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 50141 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-05-22 03:57:30,459 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1912): 253e196376ee288bc24277fe737a9a35/info is initiating minor compaction (all files) 2023-05-22 03:57:30,459 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 253e196376ee288bc24277fe737a9a35/info in TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. 2023-05-22 03:57:30,459 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/dd9050d8078445bdb89e2358a367e7df, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/ea0838ae245c4f1081b46b2fccb50e3b, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/92ca70f0c39a4769b3277dbeef6fa788] into tmpdir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.tmp, totalSize=49.0 K 2023-05-22 03:57:30,459 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting dd9050d8078445bdb89e2358a367e7df, keycount=7, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1684727848349 2023-05-22 03:57:30,460 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting ea0838ae245c4f1081b46b2fccb50e3b, keycount=18, bloomtype=ROW, size=23.7 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1684727848366 2023-05-22 03:57:30,461 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting 92ca70f0c39a4769b3277dbeef6fa788, keycount=8, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=43, earliestPutTs=1684727848401 2023-05-22 03:57:30,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.91 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.tmp/info/2dc60ca23199431f9d5d5a73bc527782 2023-05-22 03:57:30,481 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] throttle.PressureAwareThroughputController(145): 253e196376ee288bc24277fe737a9a35#info#compaction#30 average throughput is 16.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-05-22 03:57:30,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=32.0 K, regionName=253e196376ee288bc24277fe737a9a35, server=jenkins-hbase20.apache.org,43747,1684727837286 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-05-22 03:57:30,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 1.2 K connection: 148.251.75.209:54942 deadline: 1684727860483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=32.0 K, regionName=253e196376ee288bc24277fe737a9a35, server=jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:30,487 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.tmp/info/2dc60ca23199431f9d5d5a73bc527782 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/2dc60ca23199431f9d5d5a73bc527782 2023-05-22 03:57:30,494 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/2dc60ca23199431f9d5d5a73bc527782, entries=18, sequenceid=64, filesize=23.7 K 2023-05-22 03:57:30,495 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~18.91 KB/19368, heapSize ~20.48 KB/20976, currentSize=11.56 KB/11836 for 253e196376ee288bc24277fe737a9a35 in 38ms, sequenceid=64, compaction requested=false 2023-05-22 03:57:30,495 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 253e196376ee288bc24277fe737a9a35: 2023-05-22 03:57:30,496 DEBUG [MemStoreFlusher.0] regionserver.ConstantSizeRegionSplitPolicy(109): Should split because info size=72.7 K, sizeToCheck=16.0 K 2023-05-22 03:57:30,496 DEBUG [MemStoreFlusher.0] regionserver.IncreasingToUpperBoundRegionSplitPolicy(84): regionsWithCommonTable=1 2023-05-22 03:57:30,496 DEBUG [MemStoreFlusher.0] regionserver.StoreUtils(129): cannot split hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/ea0838ae245c4f1081b46b2fccb50e3b because midkey is the same as first or last row 2023-05-22 03:57:30,496 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.tmp/info/74fd33f4fcf94ccfa5f717a9288110ff as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/74fd33f4fcf94ccfa5f717a9288110ff 2023-05-22 03:57:30,503 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 253e196376ee288bc24277fe737a9a35/info of 253e196376ee288bc24277fe737a9a35 into 74fd33f4fcf94ccfa5f717a9288110ff(size=39.6 K), total size for store is 63.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-05-22 03:57:30,503 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 253e196376ee288bc24277fe737a9a35: 2023-05-22 03:57:30,503 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35., storeName=253e196376ee288bc24277fe737a9a35/info, priority=13, startTime=1684727850455; duration=0sec 2023-05-22 03:57:30,504 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.ConstantSizeRegionSplitPolicy(109): Should split because info size=63.3 K, sizeToCheck=16.0 K 2023-05-22 03:57:30,504 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.IncreasingToUpperBoundRegionSplitPolicy(84): regionsWithCommonTable=1 2023-05-22 03:57:30,504 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.StoreUtils(129): cannot split hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/74fd33f4fcf94ccfa5f717a9288110ff because midkey is the same as first or last row 2023-05-22 03:57:30,504 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:57:40,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:40,565 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 253e196376ee288bc24277fe737a9a35 1/1 column families, dataSize=12.61 KB heapSize=13.75 KB 2023-05-22 03:57:40,578 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=12.61 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.tmp/info/c304e78afbff4537b3347026e4e120ca 2023-05-22 03:57:40,583 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.tmp/info/c304e78afbff4537b3347026e4e120ca as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/c304e78afbff4537b3347026e4e120ca 2023-05-22 03:57:40,589 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/c304e78afbff4537b3347026e4e120ca, entries=12, sequenceid=80, filesize=17.4 K 2023-05-22 03:57:40,590 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~12.61 KB/12912, heapSize ~13.73 KB/14064, currentSize=1.05 KB/1076 for 253e196376ee288bc24277fe737a9a35 in 25ms, sequenceid=80, compaction requested=true 2023-05-22 03:57:40,590 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 253e196376ee288bc24277fe737a9a35: 2023-05-22 03:57:40,590 DEBUG [MemStoreFlusher.0] regionserver.ConstantSizeRegionSplitPolicy(109): Should split because info size=80.7 K, sizeToCheck=16.0 K 2023-05-22 03:57:40,590 DEBUG [MemStoreFlusher.0] regionserver.IncreasingToUpperBoundRegionSplitPolicy(84): regionsWithCommonTable=1 2023-05-22 03:57:40,590 DEBUG [MemStoreFlusher.0] regionserver.StoreUtils(129): cannot split hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/74fd33f4fcf94ccfa5f717a9288110ff because midkey is the same as first or last row 2023-05-22 03:57:40,590 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-05-22 03:57:40,590 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-05-22 03:57:40,591 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 82610 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-05-22 03:57:40,591 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1912): 253e196376ee288bc24277fe737a9a35/info is initiating minor compaction (all files) 2023-05-22 03:57:40,592 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 253e196376ee288bc24277fe737a9a35/info in TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. 2023-05-22 03:57:40,592 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/74fd33f4fcf94ccfa5f717a9288110ff, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/2dc60ca23199431f9d5d5a73bc527782, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/c304e78afbff4537b3347026e4e120ca] into tmpdir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.tmp, totalSize=80.7 K 2023-05-22 03:57:40,592 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting 74fd33f4fcf94ccfa5f717a9288110ff, keycount=33, bloomtype=ROW, size=39.6 K, encoding=NONE, compression=NONE, seqNum=43, earliestPutTs=1684727848349 2023-05-22 03:57:40,592 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting 2dc60ca23199431f9d5d5a73bc527782, keycount=18, bloomtype=ROW, size=23.7 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1684727850425 2023-05-22 03:57:40,593 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting c304e78afbff4537b3347026e4e120ca, keycount=12, bloomtype=ROW, size=17.4 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1684727850458 2023-05-22 03:57:40,604 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] throttle.PressureAwareThroughputController(145): 253e196376ee288bc24277fe737a9a35#info#compaction#32 average throughput is 21.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-05-22 03:57:40,615 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.tmp/info/f6cdba28d3a2408687565c34ef5bd2f7 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/f6cdba28d3a2408687565c34ef5bd2f7 2023-05-22 03:57:40,621 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 253e196376ee288bc24277fe737a9a35/info of 253e196376ee288bc24277fe737a9a35 into f6cdba28d3a2408687565c34ef5bd2f7(size=71.4 K), total size for store is 71.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-05-22 03:57:40,621 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 253e196376ee288bc24277fe737a9a35: 2023-05-22 03:57:40,621 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35., storeName=253e196376ee288bc24277fe737a9a35/info, priority=13, startTime=1684727860590; duration=0sec 2023-05-22 03:57:40,621 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.ConstantSizeRegionSplitPolicy(109): Should split because info size=71.4 K, sizeToCheck=16.0 K 2023-05-22 03:57:40,621 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.IncreasingToUpperBoundRegionSplitPolicy(84): regionsWithCommonTable=1 2023-05-22 03:57:40,621 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit(227): Splitting TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35., compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:57:40,622 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:57:40,622 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=43369] assignment.AssignmentManager(1140): Split request from jenkins-hbase20.apache.org,43747,1684727837286, parent={ENCODED => 253e196376ee288bc24277fe737a9a35, NAME => 'TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.', STARTKEY => '', ENDKEY => ''} splitKey=row0062 2023-05-22 03:57:40,628 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=43369] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:40,634 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=43369] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=TestLogRolling-testLogRolling, parent=253e196376ee288bc24277fe737a9a35, daughterA=39c7babe5bee422fe4aecfe6fc9ac0c6, daughterB=8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:57:40,635 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=TestLogRolling-testLogRolling, parent=253e196376ee288bc24277fe737a9a35, daughterA=39c7babe5bee422fe4aecfe6fc9ac0c6, daughterB=8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:57:40,635 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=TestLogRolling-testLogRolling, parent=253e196376ee288bc24277fe737a9a35, daughterA=39c7babe5bee422fe4aecfe6fc9ac0c6, daughterB=8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:57:40,635 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=TestLogRolling-testLogRolling, parent=253e196376ee288bc24277fe737a9a35, daughterA=39c7babe5bee422fe4aecfe6fc9ac0c6, daughterB=8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:57:40,645 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestLogRolling-testLogRolling, region=253e196376ee288bc24277fe737a9a35, UNASSIGN}] 2023-05-22 03:57:40,646 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestLogRolling-testLogRolling, region=253e196376ee288bc24277fe737a9a35, UNASSIGN 2023-05-22 03:57:40,647 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=253e196376ee288bc24277fe737a9a35, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:40,647 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1684727860647"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1684727860647"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1684727860647"}]},"ts":"1684727860647"} 2023-05-22 03:57:40,649 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 253e196376ee288bc24277fe737a9a35, server=jenkins-hbase20.apache.org,43747,1684727837286}] 2023-05-22 03:57:40,813 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(111): Close 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:40,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 253e196376ee288bc24277fe737a9a35, disabling compactions & flushes 2023-05-22 03:57:40,814 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. 2023-05-22 03:57:40,814 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. 2023-05-22 03:57:40,814 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. after waiting 0 ms 2023-05-22 03:57:40,814 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. 2023-05-22 03:57:40,814 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 253e196376ee288bc24277fe737a9a35 1/1 column families, dataSize=1.05 KB heapSize=1.38 KB 2023-05-22 03:57:40,830 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.05 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.tmp/info/df94489982d549048033da4c4ef42be3 2023-05-22 03:57:40,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.tmp/info/df94489982d549048033da4c4ef42be3 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/df94489982d549048033da4c4ef42be3 2023-05-22 03:57:40,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/df94489982d549048033da4c4ef42be3, entries=1, sequenceid=85, filesize=5.8 K 2023-05-22 03:57:40,844 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.05 KB/1076, heapSize ~1.36 KB/1392, currentSize=0 B/0 for 253e196376ee288bc24277fe737a9a35 in 30ms, sequenceid=85, compaction requested=false 2023-05-22 03:57:40,849 DEBUG [StoreCloser-TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/dd9050d8078445bdb89e2358a367e7df, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/ea0838ae245c4f1081b46b2fccb50e3b, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/74fd33f4fcf94ccfa5f717a9288110ff, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/92ca70f0c39a4769b3277dbeef6fa788, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/2dc60ca23199431f9d5d5a73bc527782, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/c304e78afbff4537b3347026e4e120ca] to archive 2023-05-22 03:57:40,850 DEBUG [StoreCloser-TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-05-22 03:57:40,852 DEBUG [StoreCloser-TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/dd9050d8078445bdb89e2358a367e7df to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/dd9050d8078445bdb89e2358a367e7df 2023-05-22 03:57:40,853 DEBUG [StoreCloser-TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/ea0838ae245c4f1081b46b2fccb50e3b to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/ea0838ae245c4f1081b46b2fccb50e3b 2023-05-22 03:57:40,854 DEBUG [StoreCloser-TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/74fd33f4fcf94ccfa5f717a9288110ff to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/74fd33f4fcf94ccfa5f717a9288110ff 2023-05-22 03:57:40,856 DEBUG [StoreCloser-TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/92ca70f0c39a4769b3277dbeef6fa788 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/92ca70f0c39a4769b3277dbeef6fa788 2023-05-22 03:57:40,857 DEBUG [StoreCloser-TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/2dc60ca23199431f9d5d5a73bc527782 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/2dc60ca23199431f9d5d5a73bc527782 2023-05-22 03:57:40,858 DEBUG [StoreCloser-TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/c304e78afbff4537b3347026e4e120ca to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/c304e78afbff4537b3347026e4e120ca 2023-05-22 03:57:40,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/recovered.edits/88.seqid, newMaxSeqId=88, maxSeqId=1 2023-05-22 03:57:40,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. 2023-05-22 03:57:40,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 253e196376ee288bc24277fe737a9a35: 2023-05-22 03:57:40,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(149): Closed 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:40,872 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=253e196376ee288bc24277fe737a9a35, regionState=CLOSED 2023-05-22 03:57:40,872 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1684727860872"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1684727860872"}]},"ts":"1684727860872"} 2023-05-22 03:57:40,876 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-05-22 03:57:40,876 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 253e196376ee288bc24277fe737a9a35, server=jenkins-hbase20.apache.org,43747,1684727837286 in 225 msec 2023-05-22 03:57:40,877 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-05-22 03:57:40,877 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestLogRolling-testLogRolling, region=253e196376ee288bc24277fe737a9a35, UNASSIGN in 231 msec 2023-05-22 03:57:40,890 INFO [PEWorker-5] assignment.SplitTableRegionProcedure(694): pid=12 splitting 2 storefiles, region=253e196376ee288bc24277fe737a9a35, threads=2 2023-05-22 03:57:40,891 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/df94489982d549048033da4c4ef42be3 for region: 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:40,891 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/f6cdba28d3a2408687565c34ef5bd2f7 for region: 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:40,900 DEBUG [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/df94489982d549048033da4c4ef42be3, top=true 2023-05-22 03:57:40,905 INFO [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/.splits/8769257a6bf880cb55211a0c78e12ae0/info/TestLogRolling-testLogRolling=253e196376ee288bc24277fe737a9a35-df94489982d549048033da4c4ef42be3 for child: 8769257a6bf880cb55211a0c78e12ae0, parent: 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:40,906 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/df94489982d549048033da4c4ef42be3 for region: 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:40,916 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/f6cdba28d3a2408687565c34ef5bd2f7 for region: 253e196376ee288bc24277fe737a9a35 2023-05-22 03:57:40,917 DEBUG [PEWorker-5] assignment.SplitTableRegionProcedure(755): pid=12 split storefiles for region 253e196376ee288bc24277fe737a9a35 Daughter A: 1 storefiles, Daughter B: 2 storefiles. 2023-05-22 03:57:40,941 DEBUG [PEWorker-5] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/39c7babe5bee422fe4aecfe6fc9ac0c6/recovered.edits/88.seqid, newMaxSeqId=88, maxSeqId=-1 2023-05-22 03:57:40,943 DEBUG [PEWorker-5] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/recovered.edits/88.seqid, newMaxSeqId=88, maxSeqId=-1 2023-05-22 03:57:40,945 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1684727860945"},{"qualifier":"splitA","vlen":70,"tag":[],"timestamp":"1684727860945"},{"qualifier":"splitB","vlen":70,"tag":[],"timestamp":"1684727860945"}]},"ts":"1684727860945"} 2023-05-22 03:57:40,945 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1684727860945"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1684727860945"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1684727860945"}]},"ts":"1684727860945"} 2023-05-22 03:57:40,945 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1684727860945"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1684727860945"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1684727860945"}]},"ts":"1684727860945"} 2023-05-22 03:57:40,988 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=43747] regionserver.HRegion(9158): Flush requested on 1588230740 2023-05-22 03:57:40,988 DEBUG [MemStoreFlusher.0] regionserver.FlushAllLargeStoresPolicy(69): Since none of the CFs were above the size, flushing all. 2023-05-22 03:57:40,989 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=4.82 KB heapSize=8.36 KB 2023-05-22 03:57:40,998 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestLogRolling-testLogRolling, region=39c7babe5bee422fe4aecfe6fc9ac0c6, ASSIGN}, {pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestLogRolling-testLogRolling, region=8769257a6bf880cb55211a0c78e12ae0, ASSIGN}] 2023-05-22 03:57:40,998 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.61 KB at sequenceid=17 (bloomFilter=false), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/.tmp/info/eb5cfd5338d5426283e359826b59e91a 2023-05-22 03:57:40,999 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestLogRolling-testLogRolling, region=8769257a6bf880cb55211a0c78e12ae0, ASSIGN 2023-05-22 03:57:40,999 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestLogRolling-testLogRolling, region=39c7babe5bee422fe4aecfe6fc9ac0c6, ASSIGN 2023-05-22 03:57:41,000 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestLogRolling-testLogRolling, region=8769257a6bf880cb55211a0c78e12ae0, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase20.apache.org,43747,1684727837286; forceNewPlan=false, retain=false 2023-05-22 03:57:41,000 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestLogRolling-testLogRolling, region=39c7babe5bee422fe4aecfe6fc9ac0c6, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase20.apache.org,43747,1684727837286; forceNewPlan=false, retain=false 2023-05-22 03:57:41,017 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=216 B at sequenceid=17 (bloomFilter=false), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/.tmp/table/66070359f29049dfac3e6ba6e5684186 2023-05-22 03:57:41,023 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/.tmp/info/eb5cfd5338d5426283e359826b59e91a as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/info/eb5cfd5338d5426283e359826b59e91a 2023-05-22 03:57:41,028 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/info/eb5cfd5338d5426283e359826b59e91a, entries=29, sequenceid=17, filesize=8.6 K 2023-05-22 03:57:41,029 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/.tmp/table/66070359f29049dfac3e6ba6e5684186 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/table/66070359f29049dfac3e6ba6e5684186 2023-05-22 03:57:41,033 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/table/66070359f29049dfac3e6ba6e5684186, entries=4, sequenceid=17, filesize=4.8 K 2023-05-22 03:57:41,034 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~4.82 KB/4939, heapSize ~8.08 KB/8272, currentSize=0 B/0 for 1588230740 in 46ms, sequenceid=17, compaction requested=false 2023-05-22 03:57:41,035 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1588230740: 2023-05-22 03:57:41,152 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=39c7babe5bee422fe4aecfe6fc9ac0c6, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:41,152 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=8769257a6bf880cb55211a0c78e12ae0, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:41,152 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1684727861151"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1684727861151"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1684727861151"}]},"ts":"1684727861151"} 2023-05-22 03:57:41,152 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1684727861151"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1684727861151"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1684727861151"}]},"ts":"1684727861151"} 2023-05-22 03:57:41,153 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=15, state=RUNNABLE; OpenRegionProcedure 39c7babe5bee422fe4aecfe6fc9ac0c6, server=jenkins-hbase20.apache.org,43747,1684727837286}] 2023-05-22 03:57:41,154 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=16, state=RUNNABLE; OpenRegionProcedure 8769257a6bf880cb55211a0c78e12ae0, server=jenkins-hbase20.apache.org,43747,1684727837286}] 2023-05-22 03:57:41,315 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:57:41,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8769257a6bf880cb55211a0c78e12ae0, NAME => 'TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.', STARTKEY => 'row0062', ENDKEY => ''} 2023-05-22 03:57:41,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestLogRolling-testLogRolling 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:57:41,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:57:41,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:57:41,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:57:41,319 INFO [StoreOpener-8769257a6bf880cb55211a0c78e12ae0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:57:41,320 DEBUG [StoreOpener-8769257a6bf880cb55211a0c78e12ae0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info 2023-05-22 03:57:41,321 DEBUG [StoreOpener-8769257a6bf880cb55211a0c78e12ae0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info 2023-05-22 03:57:41,321 INFO [StoreOpener-8769257a6bf880cb55211a0c78e12ae0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8769257a6bf880cb55211a0c78e12ae0 columnFamilyName info 2023-05-22 03:57:41,332 DEBUG [StoreOpener-8769257a6bf880cb55211a0c78e12ae0-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/TestLogRolling-testLogRolling=253e196376ee288bc24277fe737a9a35-df94489982d549048033da4c4ef42be3 2023-05-22 03:57:41,340 DEBUG [StoreOpener-8769257a6bf880cb55211a0c78e12ae0-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/f6cdba28d3a2408687565c34ef5bd2f7.253e196376ee288bc24277fe737a9a35->hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/f6cdba28d3a2408687565c34ef5bd2f7-top 2023-05-22 03:57:41,341 INFO [StoreOpener-8769257a6bf880cb55211a0c78e12ae0-1] regionserver.HStore(310): Store=8769257a6bf880cb55211a0c78e12ae0/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:57:41,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:57:41,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:57:41,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:57:41,349 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 8769257a6bf880cb55211a0c78e12ae0; next sequenceid=89; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=814705, jitterRate=0.03595103323459625}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:57:41,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:57:41,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0., pid=18, masterSystemTime=1684727861306 2023-05-22 03:57:41,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-05-22 03:57:41,351 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-05-22 03:57:41,353 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF info since it belongs to recently split daughter region TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:57:41,353 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1912): 8769257a6bf880cb55211a0c78e12ae0/info is initiating minor compaction (all files) 2023-05-22 03:57:41,353 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8769257a6bf880cb55211a0c78e12ae0/info in TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:57:41,353 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/f6cdba28d3a2408687565c34ef5bd2f7.253e196376ee288bc24277fe737a9a35->hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/f6cdba28d3a2408687565c34ef5bd2f7-top, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/TestLogRolling-testLogRolling=253e196376ee288bc24277fe737a9a35-df94489982d549048033da4c4ef42be3] into tmpdir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp, totalSize=77.2 K 2023-05-22 03:57:41,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:57:41,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:57:41,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6. 2023-05-22 03:57:41,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 39c7babe5bee422fe4aecfe6fc9ac0c6, NAME => 'TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6.', STARTKEY => '', ENDKEY => 'row0062'} 2023-05-22 03:57:41,354 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting f6cdba28d3a2408687565c34ef5bd2f7.253e196376ee288bc24277fe737a9a35, keycount=31, bloomtype=ROW, size=71.4 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1684727848349 2023-05-22 03:57:41,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestLogRolling-testLogRolling 39c7babe5bee422fe4aecfe6fc9ac0c6 2023-05-22 03:57:41,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:57:41,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 39c7babe5bee422fe4aecfe6fc9ac0c6 2023-05-22 03:57:41,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 39c7babe5bee422fe4aecfe6fc9ac0c6 2023-05-22 03:57:41,355 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting TestLogRolling-testLogRolling=253e196376ee288bc24277fe737a9a35-df94489982d549048033da4c4ef42be3, keycount=1, bloomtype=ROW, size=5.8 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1684727860566 2023-05-22 03:57:41,355 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=8769257a6bf880cb55211a0c78e12ae0, regionState=OPEN, openSeqNum=89, regionLocation=jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:41,355 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1684727861355"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1684727861355"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1684727861355"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1684727861355"}]},"ts":"1684727861355"} 2023-05-22 03:57:41,356 INFO [StoreOpener-39c7babe5bee422fe4aecfe6fc9ac0c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 39c7babe5bee422fe4aecfe6fc9ac0c6 2023-05-22 03:57:41,357 DEBUG [StoreOpener-39c7babe5bee422fe4aecfe6fc9ac0c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/39c7babe5bee422fe4aecfe6fc9ac0c6/info 2023-05-22 03:57:41,357 DEBUG [StoreOpener-39c7babe5bee422fe4aecfe6fc9ac0c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/39c7babe5bee422fe4aecfe6fc9ac0c6/info 2023-05-22 03:57:41,358 INFO [StoreOpener-39c7babe5bee422fe4aecfe6fc9ac0c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 39c7babe5bee422fe4aecfe6fc9ac0c6 columnFamilyName info 2023-05-22 03:57:41,360 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=16 2023-05-22 03:57:41,360 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=16, state=SUCCESS; OpenRegionProcedure 8769257a6bf880cb55211a0c78e12ae0, server=jenkins-hbase20.apache.org,43747,1684727837286 in 203 msec 2023-05-22 03:57:41,362 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestLogRolling-testLogRolling, region=8769257a6bf880cb55211a0c78e12ae0, ASSIGN in 362 msec 2023-05-22 03:57:41,364 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8769257a6bf880cb55211a0c78e12ae0#info#compaction#36 average throughput is 3.08 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-05-22 03:57:41,370 DEBUG [StoreOpener-39c7babe5bee422fe4aecfe6fc9ac0c6-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/39c7babe5bee422fe4aecfe6fc9ac0c6/info/f6cdba28d3a2408687565c34ef5bd2f7.253e196376ee288bc24277fe737a9a35->hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/f6cdba28d3a2408687565c34ef5bd2f7-bottom 2023-05-22 03:57:41,371 INFO [StoreOpener-39c7babe5bee422fe4aecfe6fc9ac0c6-1] regionserver.HStore(310): Store=39c7babe5bee422fe4aecfe6fc9ac0c6/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:57:41,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/39c7babe5bee422fe4aecfe6fc9ac0c6 2023-05-22 03:57:41,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/39c7babe5bee422fe4aecfe6fc9ac0c6 2023-05-22 03:57:41,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 39c7babe5bee422fe4aecfe6fc9ac0c6 2023-05-22 03:57:41,377 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 39c7babe5bee422fe4aecfe6fc9ac0c6; next sequenceid=89; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=776918, jitterRate=-0.012098848819732666}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:57:41,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 39c7babe5bee422fe4aecfe6fc9ac0c6: 2023-05-22 03:57:41,378 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6., pid=17, masterSystemTime=1684727861306 2023-05-22 03:57:41,378 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/146753cccdf8454588a3c4b8a51b9cf1 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/146753cccdf8454588a3c4b8a51b9cf1 2023-05-22 03:57:41,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-05-22 03:57:41,380 DEBUG [RS:0;jenkins-hbase20:43747-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-05-22 03:57:41,381 INFO [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF info since it belongs to recently split daughter region TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6. 2023-05-22 03:57:41,381 DEBUG [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.HStore(1912): 39c7babe5bee422fe4aecfe6fc9ac0c6/info is initiating minor compaction (all files) 2023-05-22 03:57:41,381 INFO [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 39c7babe5bee422fe4aecfe6fc9ac0c6/info in TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6. 2023-05-22 03:57:41,381 INFO [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/39c7babe5bee422fe4aecfe6fc9ac0c6/info/f6cdba28d3a2408687565c34ef5bd2f7.253e196376ee288bc24277fe737a9a35->hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/f6cdba28d3a2408687565c34ef5bd2f7-bottom] into tmpdir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/39c7babe5bee422fe4aecfe6fc9ac0c6/.tmp, totalSize=71.4 K 2023-05-22 03:57:41,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6. 2023-05-22 03:57:41,382 DEBUG [RS:0;jenkins-hbase20:43747-longCompactions-0] compactions.Compactor(207): Compacting f6cdba28d3a2408687565c34ef5bd2f7.253e196376ee288bc24277fe737a9a35, keycount=31, bloomtype=ROW, size=71.4 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1684727848349 2023-05-22 03:57:41,382 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6. 2023-05-22 03:57:41,382 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=39c7babe5bee422fe4aecfe6fc9ac0c6, regionState=OPEN, openSeqNum=89, regionLocation=jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:41,382 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1684727861382"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1684727861382"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1684727861382"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1684727861382"}]},"ts":"1684727861382"} 2023-05-22 03:57:41,387 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 2 (all) file(s) in 8769257a6bf880cb55211a0c78e12ae0/info of 8769257a6bf880cb55211a0c78e12ae0 into 146753cccdf8454588a3c4b8a51b9cf1(size=8.1 K), total size for store is 8.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-05-22 03:57:41,387 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:57:41,387 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0., storeName=8769257a6bf880cb55211a0c78e12ae0/info, priority=14, startTime=1684727861351; duration=0sec 2023-05-22 03:57:41,387 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:57:41,389 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=15 2023-05-22 03:57:41,389 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=15, state=SUCCESS; OpenRegionProcedure 39c7babe5bee422fe4aecfe6fc9ac0c6, server=jenkins-hbase20.apache.org,43747,1684727837286 in 234 msec 2023-05-22 03:57:41,390 INFO [RS:0;jenkins-hbase20:43747-longCompactions-0] throttle.PressureAwareThroughputController(145): 39c7babe5bee422fe4aecfe6fc9ac0c6#info#compaction#37 average throughput is 20.87 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-05-22 03:57:41,390 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-05-22 03:57:41,391 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestLogRolling-testLogRolling, region=39c7babe5bee422fe4aecfe6fc9ac0c6, ASSIGN in 391 msec 2023-05-22 03:57:41,392 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; SplitTableRegionProcedure table=TestLogRolling-testLogRolling, parent=253e196376ee288bc24277fe737a9a35, daughterA=39c7babe5bee422fe4aecfe6fc9ac0c6, daughterB=8769257a6bf880cb55211a0c78e12ae0 in 763 msec 2023-05-22 03:57:41,405 DEBUG [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/39c7babe5bee422fe4aecfe6fc9ac0c6/.tmp/info/f1295f955b88476286a52b19f9b2a8a4 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/39c7babe5bee422fe4aecfe6fc9ac0c6/info/f1295f955b88476286a52b19f9b2a8a4 2023-05-22 03:57:41,411 INFO [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.HStore(1652): Completed compaction of 1 (all) file(s) in 39c7babe5bee422fe4aecfe6fc9ac0c6/info of 39c7babe5bee422fe4aecfe6fc9ac0c6 into f1295f955b88476286a52b19f9b2a8a4(size=69.1 K), total size for store is 69.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-05-22 03:57:41,411 DEBUG [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 39c7babe5bee422fe4aecfe6fc9ac0c6: 2023-05-22 03:57:41,411 INFO [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6., storeName=39c7babe5bee422fe4aecfe6fc9ac0c6/info, priority=15, startTime=1684727861378; duration=0sec 2023-05-22 03:57:41,412 DEBUG [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:57:42,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 1.2 K connection: 148.251.75.209:54942 deadline: 1684727872568, exception=org.apache.hadoop.hbase.NotServingRegionException: TestLogRolling-testLogRolling,,1684727838329.253e196376ee288bc24277fe737a9a35. is not online on jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:46,432 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-05-22 03:57:52,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:57:52,687 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8769257a6bf880cb55211a0c78e12ae0 1/1 column families, dataSize=7.36 KB heapSize=8.13 KB 2023-05-22 03:57:52,695 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=7.36 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/c37b828e38fc49f8917894a078fa4b5d 2023-05-22 03:57:52,700 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/c37b828e38fc49f8917894a078fa4b5d as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c37b828e38fc49f8917894a078fa4b5d 2023-05-22 03:57:52,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=32.0 K, regionName=8769257a6bf880cb55211a0c78e12ae0, server=jenkins-hbase20.apache.org,43747,1684727837286 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-05-22 03:57:52,711 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c37b828e38fc49f8917894a078fa4b5d, entries=7, sequenceid=99, filesize=12.1 K 2023-05-22 03:57:52,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 1.2 K connection: 148.251.75.209:54942 deadline: 1684727882711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=32.0 K, regionName=8769257a6bf880cb55211a0c78e12ae0, server=jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:57:52,712 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~7.36 KB/7532, heapSize ~8.11 KB/8304, currentSize=23.12 KB/23672 for 8769257a6bf880cb55211a0c78e12ae0 in 25ms, sequenceid=99, compaction requested=false 2023-05-22 03:57:52,712 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:02,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:58:02,752 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8769257a6bf880cb55211a0c78e12ae0 1/1 column families, dataSize=24.17 KB heapSize=26.13 KB 2023-05-22 03:58:02,766 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.17 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/a071f24ee01a4ba08cd403b396e9f531 2023-05-22 03:58:02,773 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/a071f24ee01a4ba08cd403b396e9f531 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/a071f24ee01a4ba08cd403b396e9f531 2023-05-22 03:58:02,779 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/a071f24ee01a4ba08cd403b396e9f531, entries=23, sequenceid=125, filesize=29.0 K 2023-05-22 03:58:02,780 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~24.17 KB/24748, heapSize ~26.11 KB/26736, currentSize=2.10 KB/2152 for 8769257a6bf880cb55211a0c78e12ae0 in 28ms, sequenceid=125, compaction requested=true 2023-05-22 03:58:02,781 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:02,781 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:58:02,781 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-05-22 03:58:02,782 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 50299 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-05-22 03:58:02,782 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1912): 8769257a6bf880cb55211a0c78e12ae0/info is initiating minor compaction (all files) 2023-05-22 03:58:02,782 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8769257a6bf880cb55211a0c78e12ae0/info in TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:58:02,783 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/146753cccdf8454588a3c4b8a51b9cf1, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c37b828e38fc49f8917894a078fa4b5d, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/a071f24ee01a4ba08cd403b396e9f531] into tmpdir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp, totalSize=49.1 K 2023-05-22 03:58:02,783 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting 146753cccdf8454588a3c4b8a51b9cf1, keycount=3, bloomtype=ROW, size=8.1 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1684727850482 2023-05-22 03:58:02,783 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting c37b828e38fc49f8917894a078fa4b5d, keycount=7, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1684727872680 2023-05-22 03:58:02,784 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting a071f24ee01a4ba08cd403b396e9f531, keycount=23, bloomtype=ROW, size=29.0 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1684727872687 2023-05-22 03:58:02,796 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8769257a6bf880cb55211a0c78e12ae0#info#compaction#40 average throughput is 16.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-05-22 03:58:02,814 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/3aede58a22e745e0b875a014736e036e as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/3aede58a22e745e0b875a014736e036e 2023-05-22 03:58:02,821 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8769257a6bf880cb55211a0c78e12ae0/info of 8769257a6bf880cb55211a0c78e12ae0 into 3aede58a22e745e0b875a014736e036e(size=39.8 K), total size for store is 39.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-05-22 03:58:02,821 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:02,821 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0., storeName=8769257a6bf880cb55211a0c78e12ae0/info, priority=13, startTime=1684727882781; duration=0sec 2023-05-22 03:58:02,821 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:58:03,073 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-MemStoreChunkPool Statistics] regionserver.ChunkCreator$MemStoreChunkPool$StatisticsThread(426): index stats (chunk size=209715): current pool size=0, created chunk count=0, reused chunk count=0, reuseRatio=0 2023-05-22 03:58:03,073 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-MemStoreChunkPool Statistics] regionserver.ChunkCreator$MemStoreChunkPool$StatisticsThread(426): data stats (chunk size=2097152): current pool size=2, created chunk count=13, reused chunk count=32, reuseRatio=71.11% 2023-05-22 03:58:04,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:58:04,773 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8769257a6bf880cb55211a0c78e12ae0 1/1 column families, dataSize=7.36 KB heapSize=8.13 KB 2023-05-22 03:58:04,785 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=7.36 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/b198fe0434664e9aa605ef2aec3e516b 2023-05-22 03:58:04,791 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/b198fe0434664e9aa605ef2aec3e516b as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/b198fe0434664e9aa605ef2aec3e516b 2023-05-22 03:58:04,798 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/b198fe0434664e9aa605ef2aec3e516b, entries=7, sequenceid=136, filesize=12.1 K 2023-05-22 03:58:04,799 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~7.36 KB/7532, heapSize ~8.11 KB/8304, currentSize=17.86 KB/18292 for 8769257a6bf880cb55211a0c78e12ae0 in 26ms, sequenceid=136, compaction requested=false 2023-05-22 03:58:04,799 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:04,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:58:04,800 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8769257a6bf880cb55211a0c78e12ae0 1/1 column families, dataSize=18.91 KB heapSize=20.50 KB 2023-05-22 03:58:04,810 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.91 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/40036dca19884b44a540bb2e7b251972 2023-05-22 03:58:04,816 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/40036dca19884b44a540bb2e7b251972 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/40036dca19884b44a540bb2e7b251972 2023-05-22 03:58:04,824 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/40036dca19884b44a540bb2e7b251972, entries=18, sequenceid=157, filesize=23.7 K 2023-05-22 03:58:04,826 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~18.91 KB/19368, heapSize ~20.48 KB/20976, currentSize=9.46 KB/9684 for 8769257a6bf880cb55211a0c78e12ae0 in 26ms, sequenceid=157, compaction requested=true 2023-05-22 03:58:04,826 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:04,826 DEBUG [RS:0;jenkins-hbase20:43747-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-05-22 03:58:04,826 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:58:04,828 DEBUG [RS:0;jenkins-hbase20:43747-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77414 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-05-22 03:58:04,828 DEBUG [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.HStore(1912): 8769257a6bf880cb55211a0c78e12ae0/info is initiating minor compaction (all files) 2023-05-22 03:58:04,828 INFO [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8769257a6bf880cb55211a0c78e12ae0/info in TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:58:04,829 INFO [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/3aede58a22e745e0b875a014736e036e, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/b198fe0434664e9aa605ef2aec3e516b, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/40036dca19884b44a540bb2e7b251972] into tmpdir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp, totalSize=75.6 K 2023-05-22 03:58:04,829 DEBUG [RS:0;jenkins-hbase20:43747-longCompactions-0] compactions.Compactor(207): Compacting 3aede58a22e745e0b875a014736e036e, keycount=33, bloomtype=ROW, size=39.8 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1684727850482 2023-05-22 03:58:04,830 DEBUG [RS:0;jenkins-hbase20:43747-longCompactions-0] compactions.Compactor(207): Compacting b198fe0434664e9aa605ef2aec3e516b, keycount=7, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1684727882754 2023-05-22 03:58:04,830 DEBUG [RS:0;jenkins-hbase20:43747-longCompactions-0] compactions.Compactor(207): Compacting 40036dca19884b44a540bb2e7b251972, keycount=18, bloomtype=ROW, size=23.7 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1684727884774 2023-05-22 03:58:04,846 INFO [RS:0;jenkins-hbase20:43747-longCompactions-0] throttle.PressureAwareThroughputController(145): 8769257a6bf880cb55211a0c78e12ae0#info#compaction#43 average throughput is 59.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-05-22 03:58:04,876 DEBUG [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/9411e908dd464ea79bd16e7f9f65f834 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/9411e908dd464ea79bd16e7f9f65f834 2023-05-22 03:58:04,883 INFO [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8769257a6bf880cb55211a0c78e12ae0/info of 8769257a6bf880cb55211a0c78e12ae0 into 9411e908dd464ea79bd16e7f9f65f834(size=66.3 K), total size for store is 66.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-05-22 03:58:04,884 DEBUG [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:04,884 INFO [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0., storeName=8769257a6bf880cb55211a0c78e12ae0/info, priority=13, startTime=1684727884826; duration=0sec 2023-05-22 03:58:04,884 DEBUG [RS:0;jenkins-hbase20:43747-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:58:06,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:58:06,817 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8769257a6bf880cb55211a0c78e12ae0 1/1 column families, dataSize=10.51 KB heapSize=11.50 KB 2023-05-22 03:58:06,831 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=10.51 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/1df4e5bf0ff64e39903ae47eb4c31042 2023-05-22 03:58:06,838 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/1df4e5bf0ff64e39903ae47eb4c31042 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/1df4e5bf0ff64e39903ae47eb4c31042 2023-05-22 03:58:06,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/1df4e5bf0ff64e39903ae47eb4c31042, entries=10, sequenceid=171, filesize=15.3 K 2023-05-22 03:58:06,846 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~10.51 KB/10760, heapSize ~11.48 KB/11760, currentSize=16.81 KB/17216 for 8769257a6bf880cb55211a0c78e12ae0 in 29ms, sequenceid=171, compaction requested=false 2023-05-22 03:58:06,846 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:06,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:58:06,847 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8769257a6bf880cb55211a0c78e12ae0 1/1 column families, dataSize=17.86 KB heapSize=19.38 KB 2023-05-22 03:58:06,859 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.86 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/62fb5c5832a24f47baebbf9687ef8e23 2023-05-22 03:58:06,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=32.0 K, regionName=8769257a6bf880cb55211a0c78e12ae0, server=jenkins-hbase20.apache.org,43747,1684727837286 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-05-22 03:58:06,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 1.2 K connection: 148.251.75.209:54942 deadline: 1684727896861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=32.0 K, regionName=8769257a6bf880cb55211a0c78e12ae0, server=jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:58:06,865 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/62fb5c5832a24f47baebbf9687ef8e23 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/62fb5c5832a24f47baebbf9687ef8e23 2023-05-22 03:58:06,870 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/62fb5c5832a24f47baebbf9687ef8e23, entries=17, sequenceid=191, filesize=22.7 K 2023-05-22 03:58:06,871 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~17.86 KB/18292, heapSize ~19.36 KB/19824, currentSize=12.61 KB/12912 for 8769257a6bf880cb55211a0c78e12ae0 in 24ms, sequenceid=191, compaction requested=true 2023-05-22 03:58:06,871 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:06,871 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:58:06,871 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-05-22 03:58:06,873 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 106690 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-05-22 03:58:06,873 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1912): 8769257a6bf880cb55211a0c78e12ae0/info is initiating minor compaction (all files) 2023-05-22 03:58:06,873 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8769257a6bf880cb55211a0c78e12ae0/info in TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:58:06,873 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/9411e908dd464ea79bd16e7f9f65f834, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/1df4e5bf0ff64e39903ae47eb4c31042, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/62fb5c5832a24f47baebbf9687ef8e23] into tmpdir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp, totalSize=104.2 K 2023-05-22 03:58:06,873 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting 9411e908dd464ea79bd16e7f9f65f834, keycount=58, bloomtype=ROW, size=66.3 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1684727850482 2023-05-22 03:58:06,873 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting 1df4e5bf0ff64e39903ae47eb4c31042, keycount=10, bloomtype=ROW, size=15.3 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1684727884800 2023-05-22 03:58:06,874 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting 62fb5c5832a24f47baebbf9687ef8e23, keycount=17, bloomtype=ROW, size=22.7 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1684727886818 2023-05-22 03:58:06,883 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8769257a6bf880cb55211a0c78e12ae0#info#compaction#46 average throughput is 43.61 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-05-22 03:58:06,893 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/5fa957268ca8489991e60cbb39d1f60b as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/5fa957268ca8489991e60cbb39d1f60b 2023-05-22 03:58:06,899 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8769257a6bf880cb55211a0c78e12ae0/info of 8769257a6bf880cb55211a0c78e12ae0 into 5fa957268ca8489991e60cbb39d1f60b(size=94.9 K), total size for store is 94.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-05-22 03:58:06,899 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:06,899 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0., storeName=8769257a6bf880cb55211a0c78e12ae0/info, priority=13, startTime=1684727886871; duration=0sec 2023-05-22 03:58:06,899 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:58:10,115 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-05-22 03:58:16,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:58:16,896 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8769257a6bf880cb55211a0c78e12ae0 1/1 column families, dataSize=13.66 KB heapSize=14.88 KB 2023-05-22 03:58:16,904 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.66 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/b281b5fee8b247a6a13010b86131018d 2023-05-22 03:58:16,912 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/b281b5fee8b247a6a13010b86131018d as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/b281b5fee8b247a6a13010b86131018d 2023-05-22 03:58:16,916 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/b281b5fee8b247a6a13010b86131018d, entries=13, sequenceid=208, filesize=18.4 K 2023-05-22 03:58:16,917 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~13.66 KB/13988, heapSize ~14.86 KB/15216, currentSize=1.05 KB/1076 for 8769257a6bf880cb55211a0c78e12ae0 in 21ms, sequenceid=208, compaction requested=false 2023-05-22 03:58:16,917 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:18,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:58:18,913 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8769257a6bf880cb55211a0c78e12ae0 1/1 column families, dataSize=7.36 KB heapSize=8.13 KB 2023-05-22 03:58:18,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=7.36 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/48fd523b33b94ac7b141fde270a5c1d1 2023-05-22 03:58:18,930 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/48fd523b33b94ac7b141fde270a5c1d1 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/48fd523b33b94ac7b141fde270a5c1d1 2023-05-22 03:58:18,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/48fd523b33b94ac7b141fde270a5c1d1, entries=7, sequenceid=218, filesize=12.1 K 2023-05-22 03:58:18,936 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~7.36 KB/7532, heapSize ~8.11 KB/8304, currentSize=13.66 KB/13988 for 8769257a6bf880cb55211a0c78e12ae0 in 23ms, sequenceid=218, compaction requested=true 2023-05-22 03:58:18,936 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:18,936 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:58:18,936 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-05-22 03:58:18,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:58:18,937 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8769257a6bf880cb55211a0c78e12ae0 1/1 column families, dataSize=15.76 KB heapSize=17.13 KB 2023-05-22 03:58:18,937 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 128425 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-05-22 03:58:18,937 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1912): 8769257a6bf880cb55211a0c78e12ae0/info is initiating minor compaction (all files) 2023-05-22 03:58:18,937 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8769257a6bf880cb55211a0c78e12ae0/info in TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:58:18,937 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/5fa957268ca8489991e60cbb39d1f60b, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/b281b5fee8b247a6a13010b86131018d, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/48fd523b33b94ac7b141fde270a5c1d1] into tmpdir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp, totalSize=125.4 K 2023-05-22 03:58:18,938 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting 5fa957268ca8489991e60cbb39d1f60b, keycount=85, bloomtype=ROW, size=94.9 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1684727850482 2023-05-22 03:58:18,938 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting b281b5fee8b247a6a13010b86131018d, keycount=13, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1684727886847 2023-05-22 03:58:18,939 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting 48fd523b33b94ac7b141fde270a5c1d1, keycount=7, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1684727896897 2023-05-22 03:58:18,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.76 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/c6068e8f308743d99f481c7292c94c06 2023-05-22 03:58:18,961 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8769257a6bf880cb55211a0c78e12ae0#info#compaction#50 average throughput is 53.87 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-05-22 03:58:18,963 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/c6068e8f308743d99f481c7292c94c06 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c6068e8f308743d99f481c7292c94c06 2023-05-22 03:58:18,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c6068e8f308743d99f481c7292c94c06, entries=15, sequenceid=236, filesize=20.6 K 2023-05-22 03:58:18,971 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~15.76 KB/16140, heapSize ~17.11 KB/17520, currentSize=11.56 KB/11836 for 8769257a6bf880cb55211a0c78e12ae0 in 34ms, sequenceid=236, compaction requested=false 2023-05-22 03:58:18,972 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:18,972 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/c25f0c7015b944df82b09dd68ba7e916 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c25f0c7015b944df82b09dd68ba7e916 2023-05-22 03:58:18,977 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8769257a6bf880cb55211a0c78e12ae0/info of 8769257a6bf880cb55211a0c78e12ae0 into c25f0c7015b944df82b09dd68ba7e916(size=116.0 K), total size for store is 136.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-05-22 03:58:18,977 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:18,977 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0., storeName=8769257a6bf880cb55211a0c78e12ae0/info, priority=13, startTime=1684727898936; duration=0sec 2023-05-22 03:58:18,977 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:58:20,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:58:20,953 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8769257a6bf880cb55211a0c78e12ae0 1/1 column families, dataSize=12.61 KB heapSize=13.75 KB 2023-05-22 03:58:20,968 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=12.61 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/86c8bcbb3cac4bf88e6e0ce61f41ffde 2023-05-22 03:58:20,975 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/86c8bcbb3cac4bf88e6e0ce61f41ffde as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/86c8bcbb3cac4bf88e6e0ce61f41ffde 2023-05-22 03:58:20,981 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/86c8bcbb3cac4bf88e6e0ce61f41ffde, entries=12, sequenceid=252, filesize=17.4 K 2023-05-22 03:58:20,982 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~12.61 KB/12912, heapSize ~13.73 KB/14064, currentSize=17.86 KB/18292 for 8769257a6bf880cb55211a0c78e12ae0 in 28ms, sequenceid=252, compaction requested=true 2023-05-22 03:58:20,982 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:20,982 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:58:20,982 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-05-22 03:58:20,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:58:20,983 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8769257a6bf880cb55211a0c78e12ae0 1/1 column families, dataSize=18.91 KB heapSize=20.50 KB 2023-05-22 03:58:20,983 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 157637 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-05-22 03:58:20,983 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1912): 8769257a6bf880cb55211a0c78e12ae0/info is initiating minor compaction (all files) 2023-05-22 03:58:20,983 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8769257a6bf880cb55211a0c78e12ae0/info in TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:58:20,983 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c25f0c7015b944df82b09dd68ba7e916, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c6068e8f308743d99f481c7292c94c06, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/86c8bcbb3cac4bf88e6e0ce61f41ffde] into tmpdir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp, totalSize=153.9 K 2023-05-22 03:58:20,984 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting c25f0c7015b944df82b09dd68ba7e916, keycount=105, bloomtype=ROW, size=116.0 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1684727850482 2023-05-22 03:58:20,984 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting c6068e8f308743d99f481c7292c94c06, keycount=15, bloomtype=ROW, size=20.6 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1684727898914 2023-05-22 03:58:20,984 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting 86c8bcbb3cac4bf88e6e0ce61f41ffde, keycount=12, bloomtype=ROW, size=17.4 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1684727898938 2023-05-22 03:58:21,000 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.91 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/560bcc52b80946e29a323980d0a5c6fa 2023-05-22 03:58:21,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=32.0 K, regionName=8769257a6bf880cb55211a0c78e12ae0, server=jenkins-hbase20.apache.org,43747,1684727837286 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-05-22 03:58:21,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 1.2 K connection: 148.251.75.209:54942 deadline: 1684727911002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=32.0 K, regionName=8769257a6bf880cb55211a0c78e12ae0, server=jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:58:21,005 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8769257a6bf880cb55211a0c78e12ae0#info#compaction#53 average throughput is 67.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-05-22 03:58:21,007 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/560bcc52b80946e29a323980d0a5c6fa as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/560bcc52b80946e29a323980d0a5c6fa 2023-05-22 03:58:21,012 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/560bcc52b80946e29a323980d0a5c6fa, entries=18, sequenceid=273, filesize=23.7 K 2023-05-22 03:58:21,013 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~18.91 KB/19368, heapSize ~20.48 KB/20976, currentSize=11.56 KB/11836 for 8769257a6bf880cb55211a0c78e12ae0 in 30ms, sequenceid=273, compaction requested=false 2023-05-22 03:58:21,013 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:21,029 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/03c1182faedf44ca878582fd6f6cb532 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/03c1182faedf44ca878582fd6f6cb532 2023-05-22 03:58:21,035 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8769257a6bf880cb55211a0c78e12ae0/info of 8769257a6bf880cb55211a0c78e12ae0 into 03c1182faedf44ca878582fd6f6cb532(size=144.7 K), total size for store is 168.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-05-22 03:58:21,035 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:21,035 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0., storeName=8769257a6bf880cb55211a0c78e12ae0/info, priority=13, startTime=1684727900982; duration=0sec 2023-05-22 03:58:21,035 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:58:31,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:58:31,102 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8769257a6bf880cb55211a0c78e12ae0 1/1 column families, dataSize=12.61 KB heapSize=13.75 KB 2023-05-22 03:58:31,115 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=12.61 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/c4e6fc4137f743cc8b0f6d26399f33a3 2023-05-22 03:58:31,122 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/c4e6fc4137f743cc8b0f6d26399f33a3 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c4e6fc4137f743cc8b0f6d26399f33a3 2023-05-22 03:58:31,127 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c4e6fc4137f743cc8b0f6d26399f33a3, entries=12, sequenceid=289, filesize=17.4 K 2023-05-22 03:58:31,128 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~12.61 KB/12912, heapSize ~13.73 KB/14064, currentSize=1.05 KB/1076 for 8769257a6bf880cb55211a0c78e12ae0 in 26ms, sequenceid=289, compaction requested=true 2023-05-22 03:58:31,128 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:31,129 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:58:31,129 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-05-22 03:58:31,130 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 190316 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-05-22 03:58:31,130 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1912): 8769257a6bf880cb55211a0c78e12ae0/info is initiating minor compaction (all files) 2023-05-22 03:58:31,130 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8769257a6bf880cb55211a0c78e12ae0/info in TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:58:31,130 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/03c1182faedf44ca878582fd6f6cb532, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/560bcc52b80946e29a323980d0a5c6fa, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c4e6fc4137f743cc8b0f6d26399f33a3] into tmpdir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp, totalSize=185.9 K 2023-05-22 03:58:31,130 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting 03c1182faedf44ca878582fd6f6cb532, keycount=132, bloomtype=ROW, size=144.7 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1684727850482 2023-05-22 03:58:31,131 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting 560bcc52b80946e29a323980d0a5c6fa, keycount=18, bloomtype=ROW, size=23.7 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1684727900955 2023-05-22 03:58:31,131 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting c4e6fc4137f743cc8b0f6d26399f33a3, keycount=12, bloomtype=ROW, size=17.4 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1684727900983 2023-05-22 03:58:31,141 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8769257a6bf880cb55211a0c78e12ae0#info#compaction#55 average throughput is 83.12 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-05-22 03:58:31,153 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/c3f1885c9bb84c3f862bf6920bc94bb1 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c3f1885c9bb84c3f862bf6920bc94bb1 2023-05-22 03:58:31,158 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8769257a6bf880cb55211a0c78e12ae0/info of 8769257a6bf880cb55211a0c78e12ae0 into c3f1885c9bb84c3f862bf6920bc94bb1(size=176.5 K), total size for store is 176.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-05-22 03:58:31,158 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:31,158 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0., storeName=8769257a6bf880cb55211a0c78e12ae0/info, priority=13, startTime=1684727911129; duration=0sec 2023-05-22 03:58:31,158 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:58:33,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:58:33,123 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8769257a6bf880cb55211a0c78e12ae0 1/1 column families, dataSize=7.36 KB heapSize=8.13 KB 2023-05-22 03:58:33,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=32.0 K, regionName=8769257a6bf880cb55211a0c78e12ae0, server=jenkins-hbase20.apache.org,43747,1684727837286 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-05-22 03:58:33,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 1.2 K connection: 148.251.75.209:54942 deadline: 1684727923146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=32.0 K, regionName=8769257a6bf880cb55211a0c78e12ae0, server=jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:58:33,534 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=7.36 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/0169fda2acc446cdbd9b89002d638716 2023-05-22 03:58:33,548 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/0169fda2acc446cdbd9b89002d638716 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/0169fda2acc446cdbd9b89002d638716 2023-05-22 03:58:33,556 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/0169fda2acc446cdbd9b89002d638716, entries=7, sequenceid=300, filesize=12.1 K 2023-05-22 03:58:33,557 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~7.36 KB/7532, heapSize ~8.11 KB/8304, currentSize=23.12 KB/23672 for 8769257a6bf880cb55211a0c78e12ae0 in 434ms, sequenceid=300, compaction requested=false 2023-05-22 03:58:33,557 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:43,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43747] regionserver.HRegion(9158): Flush requested on 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:58:43,181 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8769257a6bf880cb55211a0c78e12ae0 1/1 column families, dataSize=24.17 KB heapSize=26.13 KB 2023-05-22 03:58:43,190 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.17 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/dbcde436c8f9464a9a3a1e286a298968 2023-05-22 03:58:43,197 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/dbcde436c8f9464a9a3a1e286a298968 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/dbcde436c8f9464a9a3a1e286a298968 2023-05-22 03:58:43,203 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/dbcde436c8f9464a9a3a1e286a298968, entries=23, sequenceid=326, filesize=29.0 K 2023-05-22 03:58:43,205 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~24.17 KB/24748, heapSize ~26.11 KB/26736, currentSize=3.15 KB/3228 for 8769257a6bf880cb55211a0c78e12ae0 in 24ms, sequenceid=326, compaction requested=true 2023-05-22 03:58:43,205 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:43,205 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-05-22 03:58:43,205 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-05-22 03:58:43,207 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 222796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-05-22 03:58:43,207 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1912): 8769257a6bf880cb55211a0c78e12ae0/info is initiating minor compaction (all files) 2023-05-22 03:58:43,207 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8769257a6bf880cb55211a0c78e12ae0/info in TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:58:43,207 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c3f1885c9bb84c3f862bf6920bc94bb1, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/0169fda2acc446cdbd9b89002d638716, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/dbcde436c8f9464a9a3a1e286a298968] into tmpdir=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp, totalSize=217.6 K 2023-05-22 03:58:43,208 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting c3f1885c9bb84c3f862bf6920bc94bb1, keycount=162, bloomtype=ROW, size=176.5 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1684727850482 2023-05-22 03:58:43,208 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting 0169fda2acc446cdbd9b89002d638716, keycount=7, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1684727911103 2023-05-22 03:58:43,209 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] compactions.Compactor(207): Compacting dbcde436c8f9464a9a3a1e286a298968, keycount=23, bloomtype=ROW, size=29.0 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1684727913124 2023-05-22 03:58:43,219 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8769257a6bf880cb55211a0c78e12ae0#info#compaction#58 average throughput is 98.51 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-05-22 03:58:43,233 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/9086876d5f1f4cbb8cbadc7ee483f282 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/9086876d5f1f4cbb8cbadc7ee483f282 2023-05-22 03:58:43,294 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8769257a6bf880cb55211a0c78e12ae0/info of 8769257a6bf880cb55211a0c78e12ae0 into 9086876d5f1f4cbb8cbadc7ee483f282(size=208.2 K), total size for store is 208.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-05-22 03:58:43,294 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:43,294 INFO [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0., storeName=8769257a6bf880cb55211a0c78e12ae0/info, priority=13, startTime=1684727923205; duration=0sec 2023-05-22 03:58:43,294 DEBUG [RS:0;jenkins-hbase20:43747-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-05-22 03:58:45,185 INFO [Listener at localhost.localdomain/33915] wal.AbstractTestLogRolling(188): after writing there are 0 log files 2023-05-22 03:58:45,215 INFO [Listener at localhost.localdomain/33915] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/WALs/jenkins-hbase20.apache.org,43747,1684727837286/jenkins-hbase20.apache.org%2C43747%2C1684727837286.1684727837664 with entries=314, filesize=308.54 KB; new WAL /user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/WALs/jenkins-hbase20.apache.org,43747,1684727837286/jenkins-hbase20.apache.org%2C43747%2C1684727837286.1684727925186 2023-05-22 03:58:45,216 DEBUG [Listener at localhost.localdomain/33915] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39185,DS-c8310352-76ac-432e-a8a9-39ce69f5cab1,DISK], DatanodeInfoWithStorage[127.0.0.1:46555,DS-2d15a1ed-d313-4888-a97e-9e66f2a56afd,DISK]] 2023-05-22 03:58:45,216 DEBUG [Listener at localhost.localdomain/33915] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/WALs/jenkins-hbase20.apache.org,43747,1684727837286/jenkins-hbase20.apache.org%2C43747%2C1684727837286.1684727837664 is not closed yet, will try archiving it next time 2023-05-22 03:58:45,223 INFO [Listener at localhost.localdomain/33915] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=2.26 KB heapSize=4.19 KB 2023-05-22 03:58:45,233 INFO [Listener at localhost.localdomain/33915] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.26 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/.tmp/info/955ae0625c294766a8e4a00a65a9363d 2023-05-22 03:58:45,240 DEBUG [Listener at localhost.localdomain/33915] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/.tmp/info/955ae0625c294766a8e4a00a65a9363d as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/info/955ae0625c294766a8e4a00a65a9363d 2023-05-22 03:58:45,247 INFO [Listener at localhost.localdomain/33915] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/info/955ae0625c294766a8e4a00a65a9363d, entries=16, sequenceid=24, filesize=7.0 K 2023-05-22 03:58:45,248 INFO [Listener at localhost.localdomain/33915] regionserver.HRegion(2948): Finished flush of dataSize ~2.26 KB/2316, heapSize ~3.67 KB/3760, currentSize=0 B/0 for 1588230740 in 25ms, sequenceid=24, compaction requested=false 2023-05-22 03:58:45,248 DEBUG [Listener at localhost.localdomain/33915] regionserver.HRegion(2446): Flush status journal for 1588230740: 2023-05-22 03:58:45,248 INFO [Listener at localhost.localdomain/33915] regionserver.HRegion(2745): Flushing 89fa14d6dab8af32afc99240e7dd087f 1/1 column families, dataSize=78 B heapSize=488 B 2023-05-22 03:58:45,261 INFO [Listener at localhost.localdomain/33915] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/namespace/89fa14d6dab8af32afc99240e7dd087f/.tmp/info/e329c16348ea4d79a3c5dd7af3ea32fc 2023-05-22 03:58:45,269 DEBUG [Listener at localhost.localdomain/33915] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/namespace/89fa14d6dab8af32afc99240e7dd087f/.tmp/info/e329c16348ea4d79a3c5dd7af3ea32fc as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/namespace/89fa14d6dab8af32afc99240e7dd087f/info/e329c16348ea4d79a3c5dd7af3ea32fc 2023-05-22 03:58:45,276 INFO [Listener at localhost.localdomain/33915] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/namespace/89fa14d6dab8af32afc99240e7dd087f/info/e329c16348ea4d79a3c5dd7af3ea32fc, entries=2, sequenceid=6, filesize=4.8 K 2023-05-22 03:58:45,277 INFO [Listener at localhost.localdomain/33915] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 89fa14d6dab8af32afc99240e7dd087f in 29ms, sequenceid=6, compaction requested=false 2023-05-22 03:58:45,278 DEBUG [Listener at localhost.localdomain/33915] regionserver.HRegion(2446): Flush status journal for 89fa14d6dab8af32afc99240e7dd087f: 2023-05-22 03:58:45,278 INFO [Listener at localhost.localdomain/33915] regionserver.HRegion(2745): Flushing 8769257a6bf880cb55211a0c78e12ae0 1/1 column families, dataSize=3.15 KB heapSize=3.63 KB 2023-05-22 03:58:45,288 INFO [Listener at localhost.localdomain/33915] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.15 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/4d8858582b1e47c9b2f0a19b7951462a 2023-05-22 03:58:45,297 DEBUG [Listener at localhost.localdomain/33915] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/.tmp/info/4d8858582b1e47c9b2f0a19b7951462a as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/4d8858582b1e47c9b2f0a19b7951462a 2023-05-22 03:58:45,302 INFO [Listener at localhost.localdomain/33915] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/4d8858582b1e47c9b2f0a19b7951462a, entries=3, sequenceid=333, filesize=7.9 K 2023-05-22 03:58:45,303 INFO [Listener at localhost.localdomain/33915] regionserver.HRegion(2948): Finished flush of dataSize ~3.15 KB/3228, heapSize ~3.61 KB/3696, currentSize=0 B/0 for 8769257a6bf880cb55211a0c78e12ae0 in 25ms, sequenceid=333, compaction requested=false 2023-05-22 03:58:45,303 DEBUG [Listener at localhost.localdomain/33915] regionserver.HRegion(2446): Flush status journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:45,303 DEBUG [Listener at localhost.localdomain/33915] regionserver.HRegion(2446): Flush status journal for 39c7babe5bee422fe4aecfe6fc9ac0c6: 2023-05-22 03:58:45,313 INFO [Listener at localhost.localdomain/33915] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/WALs/jenkins-hbase20.apache.org,43747,1684727837286/jenkins-hbase20.apache.org%2C43747%2C1684727837286.1684727925186 with entries=4, filesize=1.22 KB; new WAL /user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/WALs/jenkins-hbase20.apache.org,43747,1684727837286/jenkins-hbase20.apache.org%2C43747%2C1684727837286.1684727925303 2023-05-22 03:58:45,313 DEBUG [Listener at localhost.localdomain/33915] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39185,DS-c8310352-76ac-432e-a8a9-39ce69f5cab1,DISK], DatanodeInfoWithStorage[127.0.0.1:46555,DS-2d15a1ed-d313-4888-a97e-9e66f2a56afd,DISK]] 2023-05-22 03:58:45,313 DEBUG [Listener at localhost.localdomain/33915] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/WALs/jenkins-hbase20.apache.org,43747,1684727837286/jenkins-hbase20.apache.org%2C43747%2C1684727837286.1684727925186 is not closed yet, will try archiving it next time 2023-05-22 03:58:45,314 INFO [WAL-Archive-0] wal.AbstractFSWAL(783): Archiving hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/WALs/jenkins-hbase20.apache.org,43747,1684727837286/jenkins-hbase20.apache.org%2C43747%2C1684727837286.1684727837664 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/oldWALs/jenkins-hbase20.apache.org%2C43747%2C1684727837286.1684727837664 2023-05-22 03:58:45,315 INFO [Listener at localhost.localdomain/33915] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1]) 2023-05-22 03:58:45,317 INFO [WAL-Archive-0] wal.AbstractFSWAL(783): Archiving hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/WALs/jenkins-hbase20.apache.org,43747,1684727837286/jenkins-hbase20.apache.org%2C43747%2C1684727837286.1684727925186 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/oldWALs/jenkins-hbase20.apache.org%2C43747%2C1684727837286.1684727925186 2023-05-22 03:58:45,416 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-05-22 03:58:45,416 INFO [Listener at localhost.localdomain/33915] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-05-22 03:58:45,416 DEBUG [Listener at localhost.localdomain/33915] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78d0f774 to 127.0.0.1:59657 2023-05-22 03:58:45,416 DEBUG [Listener at localhost.localdomain/33915] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:58:45,417 DEBUG [Listener at localhost.localdomain/33915] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-05-22 03:58:45,417 DEBUG [Listener at localhost.localdomain/33915] util.JVMClusterUtil(257): Found active master hash=624950227, stopped=false 2023-05-22 03:58:45,417 INFO [Listener at localhost.localdomain/33915] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,43369,1684727837208 2023-05-22 03:58:45,419 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-05-22 03:58:45,420 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): regionserver:43747-0x1017150ca2d0001, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-05-22 03:58:45,420 INFO [Listener at localhost.localdomain/33915] procedure2.ProcedureExecutor(629): Stopping 2023-05-22 03:58:45,420 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:58:45,420 DEBUG [Listener at localhost.localdomain/33915] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x579aa482 to 127.0.0.1:59657 2023-05-22 03:58:45,421 DEBUG [Listener at localhost.localdomain/33915] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:58:45,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:58:45,422 INFO [Listener at localhost.localdomain/33915] regionserver.HRegionServer(2295): ***** STOPPING region server 'jenkins-hbase20.apache.org,43747,1684727837286' ***** 2023-05-22 03:58:45,422 INFO [Listener at localhost.localdomain/33915] regionserver.HRegionServer(2309): STOPPED: Shutdown requested 2023-05-22 03:58:45,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43747-0x1017150ca2d0001, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:58:45,422 INFO [RS:0;jenkins-hbase20:43747] regionserver.HeapMemoryManager(220): Stopping 2023-05-22 03:58:45,422 INFO [RS:0;jenkins-hbase20:43747] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-05-22 03:58:45,423 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-05-22 03:58:45,423 INFO [RS:0;jenkins-hbase20:43747] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-05-22 03:58:45,423 INFO [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(3303): Received CLOSE for 89fa14d6dab8af32afc99240e7dd087f 2023-05-22 03:58:45,423 INFO [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(3303): Received CLOSE for 8769257a6bf880cb55211a0c78e12ae0 2023-05-22 03:58:45,423 INFO [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(3303): Received CLOSE for 39c7babe5bee422fe4aecfe6fc9ac0c6 2023-05-22 03:58:45,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 89fa14d6dab8af32afc99240e7dd087f, disabling compactions & flushes 2023-05-22 03:58:45,423 INFO [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:58:45,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f. 2023-05-22 03:58:45,424 DEBUG [RS:0;jenkins-hbase20:43747] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13a975ce to 127.0.0.1:59657 2023-05-22 03:58:45,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f. 2023-05-22 03:58:45,424 DEBUG [RS:0;jenkins-hbase20:43747] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:58:45,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f. after waiting 0 ms 2023-05-22 03:58:45,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f. 2023-05-22 03:58:45,424 INFO [RS:0;jenkins-hbase20:43747] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-05-22 03:58:45,424 INFO [RS:0;jenkins-hbase20:43747] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-05-22 03:58:45,424 INFO [RS:0;jenkins-hbase20:43747] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-05-22 03:58:45,424 INFO [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(3303): Received CLOSE for 1588230740 2023-05-22 03:58:45,426 INFO [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(1474): Waiting on 4 regions to close 2023-05-22 03:58:45,426 DEBUG [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, 89fa14d6dab8af32afc99240e7dd087f=hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f., 8769257a6bf880cb55211a0c78e12ae0=TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0., 39c7babe5bee422fe4aecfe6fc9ac0c6=TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6.} 2023-05-22 03:58:45,426 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-05-22 03:58:45,426 DEBUG [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(1504): Waiting on 1588230740, 39c7babe5bee422fe4aecfe6fc9ac0c6, 8769257a6bf880cb55211a0c78e12ae0, 89fa14d6dab8af32afc99240e7dd087f 2023-05-22 03:58:45,426 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-05-22 03:58:45,427 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-05-22 03:58:45,427 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-05-22 03:58:45,428 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-05-22 03:58:45,441 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/namespace/89fa14d6dab8af32afc99240e7dd087f/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-05-22 03:58:45,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f. 2023-05-22 03:58:45,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 89fa14d6dab8af32afc99240e7dd087f: 2023-05-22 03:58:45,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1684727837832.89fa14d6dab8af32afc99240e7dd087f. 2023-05-22 03:58:45,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 8769257a6bf880cb55211a0c78e12ae0, disabling compactions & flushes 2023-05-22 03:58:45,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:58:45,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:58:45,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. after waiting 0 ms 2023-05-22 03:58:45,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:58:45,457 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=1 2023-05-22 03:58:45,466 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-05-22 03:58:45,466 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-05-22 03:58:45,466 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-05-22 03:58:45,466 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-05-22 03:58:45,467 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/f6cdba28d3a2408687565c34ef5bd2f7.253e196376ee288bc24277fe737a9a35->hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/f6cdba28d3a2408687565c34ef5bd2f7-top, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/146753cccdf8454588a3c4b8a51b9cf1, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/TestLogRolling-testLogRolling=253e196376ee288bc24277fe737a9a35-df94489982d549048033da4c4ef42be3, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c37b828e38fc49f8917894a078fa4b5d, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/3aede58a22e745e0b875a014736e036e, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/a071f24ee01a4ba08cd403b396e9f531, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/b198fe0434664e9aa605ef2aec3e516b, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/9411e908dd464ea79bd16e7f9f65f834, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/40036dca19884b44a540bb2e7b251972, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/1df4e5bf0ff64e39903ae47eb4c31042, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/5fa957268ca8489991e60cbb39d1f60b, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/62fb5c5832a24f47baebbf9687ef8e23, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/b281b5fee8b247a6a13010b86131018d, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c25f0c7015b944df82b09dd68ba7e916, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/48fd523b33b94ac7b141fde270a5c1d1, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c6068e8f308743d99f481c7292c94c06, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/03c1182faedf44ca878582fd6f6cb532, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/86c8bcbb3cac4bf88e6e0ce61f41ffde, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/560bcc52b80946e29a323980d0a5c6fa, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c3f1885c9bb84c3f862bf6920bc94bb1, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c4e6fc4137f743cc8b0f6d26399f33a3, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/0169fda2acc446cdbd9b89002d638716, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/dbcde436c8f9464a9a3a1e286a298968] to archive 2023-05-22 03:58:45,467 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-05-22 03:58:45,469 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/f6cdba28d3a2408687565c34ef5bd2f7.253e196376ee288bc24277fe737a9a35 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/f6cdba28d3a2408687565c34ef5bd2f7.253e196376ee288bc24277fe737a9a35 2023-05-22 03:58:45,470 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/146753cccdf8454588a3c4b8a51b9cf1 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/146753cccdf8454588a3c4b8a51b9cf1 2023-05-22 03:58:45,471 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/TestLogRolling-testLogRolling=253e196376ee288bc24277fe737a9a35-df94489982d549048033da4c4ef42be3 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/TestLogRolling-testLogRolling=253e196376ee288bc24277fe737a9a35-df94489982d549048033da4c4ef42be3 2023-05-22 03:58:45,472 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c37b828e38fc49f8917894a078fa4b5d to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c37b828e38fc49f8917894a078fa4b5d 2023-05-22 03:58:45,473 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/3aede58a22e745e0b875a014736e036e to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/3aede58a22e745e0b875a014736e036e 2023-05-22 03:58:45,474 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/a071f24ee01a4ba08cd403b396e9f531 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/a071f24ee01a4ba08cd403b396e9f531 2023-05-22 03:58:45,475 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/b198fe0434664e9aa605ef2aec3e516b to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/b198fe0434664e9aa605ef2aec3e516b 2023-05-22 03:58:45,476 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/9411e908dd464ea79bd16e7f9f65f834 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/9411e908dd464ea79bd16e7f9f65f834 2023-05-22 03:58:45,477 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/40036dca19884b44a540bb2e7b251972 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/40036dca19884b44a540bb2e7b251972 2023-05-22 03:58:45,478 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/1df4e5bf0ff64e39903ae47eb4c31042 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/1df4e5bf0ff64e39903ae47eb4c31042 2023-05-22 03:58:45,479 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/5fa957268ca8489991e60cbb39d1f60b to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/5fa957268ca8489991e60cbb39d1f60b 2023-05-22 03:58:45,480 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/62fb5c5832a24f47baebbf9687ef8e23 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/62fb5c5832a24f47baebbf9687ef8e23 2023-05-22 03:58:45,481 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/b281b5fee8b247a6a13010b86131018d to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/b281b5fee8b247a6a13010b86131018d 2023-05-22 03:58:45,482 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c25f0c7015b944df82b09dd68ba7e916 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c25f0c7015b944df82b09dd68ba7e916 2023-05-22 03:58:45,483 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/48fd523b33b94ac7b141fde270a5c1d1 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/48fd523b33b94ac7b141fde270a5c1d1 2023-05-22 03:58:45,484 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c6068e8f308743d99f481c7292c94c06 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c6068e8f308743d99f481c7292c94c06 2023-05-22 03:58:45,485 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/03c1182faedf44ca878582fd6f6cb532 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/03c1182faedf44ca878582fd6f6cb532 2023-05-22 03:58:45,486 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/86c8bcbb3cac4bf88e6e0ce61f41ffde to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/86c8bcbb3cac4bf88e6e0ce61f41ffde 2023-05-22 03:58:45,488 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/560bcc52b80946e29a323980d0a5c6fa to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/560bcc52b80946e29a323980d0a5c6fa 2023-05-22 03:58:45,489 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c3f1885c9bb84c3f862bf6920bc94bb1 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c3f1885c9bb84c3f862bf6920bc94bb1 2023-05-22 03:58:45,489 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c4e6fc4137f743cc8b0f6d26399f33a3 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/c4e6fc4137f743cc8b0f6d26399f33a3 2023-05-22 03:58:45,491 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/0169fda2acc446cdbd9b89002d638716 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/0169fda2acc446cdbd9b89002d638716 2023-05-22 03:58:45,492 DEBUG [StoreCloser-TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/dbcde436c8f9464a9a3a1e286a298968 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/info/dbcde436c8f9464a9a3a1e286a298968 2023-05-22 03:58:45,497 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/8769257a6bf880cb55211a0c78e12ae0/recovered.edits/336.seqid, newMaxSeqId=336, maxSeqId=88 2023-05-22 03:58:45,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:58:45,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 8769257a6bf880cb55211a0c78e12ae0: 2023-05-22 03:58:45,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed TestLogRolling-testLogRolling,row0062,1684727860628.8769257a6bf880cb55211a0c78e12ae0. 2023-05-22 03:58:45,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 39c7babe5bee422fe4aecfe6fc9ac0c6, disabling compactions & flushes 2023-05-22 03:58:45,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6. 2023-05-22 03:58:45,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6. 2023-05-22 03:58:45,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6. after waiting 0 ms 2023-05-22 03:58:45,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6. 2023-05-22 03:58:45,502 DEBUG [StoreCloser-TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/39c7babe5bee422fe4aecfe6fc9ac0c6/info/f6cdba28d3a2408687565c34ef5bd2f7.253e196376ee288bc24277fe737a9a35->hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/253e196376ee288bc24277fe737a9a35/info/f6cdba28d3a2408687565c34ef5bd2f7-bottom] to archive 2023-05-22 03:58:45,503 DEBUG [StoreCloser-TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-05-22 03:58:45,505 DEBUG [StoreCloser-TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/39c7babe5bee422fe4aecfe6fc9ac0c6/info/f6cdba28d3a2408687565c34ef5bd2f7.253e196376ee288bc24277fe737a9a35 to hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/archive/data/default/TestLogRolling-testLogRolling/39c7babe5bee422fe4aecfe6fc9ac0c6/info/f6cdba28d3a2408687565c34ef5bd2f7.253e196376ee288bc24277fe737a9a35 2023-05-22 03:58:45,513 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/data/default/TestLogRolling-testLogRolling/39c7babe5bee422fe4aecfe6fc9ac0c6/recovered.edits/93.seqid, newMaxSeqId=93, maxSeqId=88 2023-05-22 03:58:45,515 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6. 2023-05-22 03:58:45,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 39c7babe5bee422fe4aecfe6fc9ac0c6: 2023-05-22 03:58:45,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed TestLogRolling-testLogRolling,,1684727860628.39c7babe5bee422fe4aecfe6fc9ac0c6. 2023-05-22 03:58:45,540 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-05-22 03:58:45,541 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-05-22 03:58:45,546 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-05-22 03:58:45,627 INFO [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,43747,1684727837286; all regions closed. 2023-05-22 03:58:45,627 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/WALs/jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:58:45,633 DEBUG [RS:0;jenkins-hbase20:43747] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/oldWALs 2023-05-22 03:58:45,633 INFO [RS:0;jenkins-hbase20:43747] wal.AbstractFSWAL(1031): Closed WAL: FSHLog jenkins-hbase20.apache.org%2C43747%2C1684727837286.meta:.meta(num 1684727837781) 2023-05-22 03:58:45,633 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/WALs/jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:58:45,641 DEBUG [RS:0;jenkins-hbase20:43747] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/oldWALs 2023-05-22 03:58:45,641 INFO [RS:0;jenkins-hbase20:43747] wal.AbstractFSWAL(1031): Closed WAL: FSHLog jenkins-hbase20.apache.org%2C43747%2C1684727837286:(num 1684727925303) 2023-05-22 03:58:45,641 DEBUG [RS:0;jenkins-hbase20:43747] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:58:45,641 INFO [RS:0;jenkins-hbase20:43747] regionserver.LeaseManager(133): Closed leases 2023-05-22 03:58:45,642 INFO [RS:0;jenkins-hbase20:43747] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-05-22 03:58:45,642 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-05-22 03:58:45,642 INFO [RS:0;jenkins-hbase20:43747] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:43747 2023-05-22 03:58:45,644 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): regionserver:43747-0x1017150ca2d0001, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,43747,1684727837286 2023-05-22 03:58:45,644 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:58:45,644 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): regionserver:43747-0x1017150ca2d0001, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:58:45,644 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,43747,1684727837286] 2023-05-22 03:58:45,644 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,43747,1684727837286; numProcessing=1 2023-05-22 03:58:45,645 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,43747,1684727837286 already deleted, retry=false 2023-05-22 03:58:45,645 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,43747,1684727837286 expired; onlineServers=0 2023-05-22 03:58:45,645 INFO [RegionServerTracker-0] regionserver.HRegionServer(2295): ***** STOPPING region server 'jenkins-hbase20.apache.org,43369,1684727837208' ***** 2023-05-22 03:58:45,645 INFO [RegionServerTracker-0] regionserver.HRegionServer(2309): STOPPED: Cluster shutdown set; onlineServer=0 2023-05-22 03:58:45,646 DEBUG [M:0;jenkins-hbase20:43369] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21007bac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-05-22 03:58:45,646 INFO [M:0;jenkins-hbase20:43369] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,43369,1684727837208 2023-05-22 03:58:45,646 INFO [M:0;jenkins-hbase20:43369] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,43369,1684727837208; all regions closed. 2023-05-22 03:58:45,646 DEBUG [M:0;jenkins-hbase20:43369] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:58:45,646 DEBUG [M:0;jenkins-hbase20:43369] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-05-22 03:58:45,646 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-05-22 03:58:45,646 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727837415] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727837415,5,FailOnTimeoutGroup] 2023-05-22 03:58:45,646 DEBUG [M:0;jenkins-hbase20:43369] cleaner.HFileCleaner(317): Stopping file delete threads 2023-05-22 03:58:45,646 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727837415] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727837415,5,FailOnTimeoutGroup] 2023-05-22 03:58:45,647 INFO [M:0;jenkins-hbase20:43369] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-05-22 03:58:45,647 INFO [M:0;jenkins-hbase20:43369] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-05-22 03:58:45,647 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-05-22 03:58:45,647 INFO [M:0;jenkins-hbase20:43369] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-05-22 03:58:45,647 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:58:45,647 DEBUG [M:0;jenkins-hbase20:43369] master.HMaster(1512): Stopping service threads 2023-05-22 03:58:45,647 INFO [M:0;jenkins-hbase20:43369] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-05-22 03:58:45,648 ERROR [M:0;jenkins-hbase20:43369] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-05-22 03:58:45,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:58:45,648 INFO [M:0;jenkins-hbase20:43369] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-05-22 03:58:45,648 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-05-22 03:58:45,648 DEBUG [M:0;jenkins-hbase20:43369] zookeeper.ZKUtil(398): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-05-22 03:58:45,648 WARN [M:0;jenkins-hbase20:43369] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-05-22 03:58:45,648 INFO [M:0;jenkins-hbase20:43369] assignment.AssignmentManager(315): Stopping assignment manager 2023-05-22 03:58:45,648 INFO [M:0;jenkins-hbase20:43369] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-05-22 03:58:45,648 DEBUG [M:0;jenkins-hbase20:43369] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-05-22 03:58:45,648 INFO [M:0;jenkins-hbase20:43369] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:58:45,649 DEBUG [M:0;jenkins-hbase20:43369] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:58:45,649 DEBUG [M:0;jenkins-hbase20:43369] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-05-22 03:58:45,649 DEBUG [M:0;jenkins-hbase20:43369] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:58:45,649 INFO [M:0;jenkins-hbase20:43369] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=64.78 KB heapSize=78.52 KB 2023-05-22 03:58:45,659 INFO [M:0;jenkins-hbase20:43369] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.78 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f747f2c328504f7e83e391a96f18ddd2 2023-05-22 03:58:45,664 INFO [M:0;jenkins-hbase20:43369] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f747f2c328504f7e83e391a96f18ddd2 2023-05-22 03:58:45,666 DEBUG [M:0;jenkins-hbase20:43369] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f747f2c328504f7e83e391a96f18ddd2 as hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f747f2c328504f7e83e391a96f18ddd2 2023-05-22 03:58:45,670 INFO [M:0;jenkins-hbase20:43369] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f747f2c328504f7e83e391a96f18ddd2 2023-05-22 03:58:45,670 INFO [M:0;jenkins-hbase20:43369] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45435/user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f747f2c328504f7e83e391a96f18ddd2, entries=18, sequenceid=160, filesize=6.9 K 2023-05-22 03:58:45,671 INFO [M:0;jenkins-hbase20:43369] regionserver.HRegion(2948): Finished flush of dataSize ~64.78 KB/66332, heapSize ~78.51 KB/80392, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 22ms, sequenceid=160, compaction requested=false 2023-05-22 03:58:45,673 INFO [M:0;jenkins-hbase20:43369] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:58:45,673 DEBUG [M:0;jenkins-hbase20:43369] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:58:45,673 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/7b2148b9-84ca-6f92-6895-fe53d3608d24/MasterData/WALs/jenkins-hbase20.apache.org,43369,1684727837208 2023-05-22 03:58:45,678 INFO [M:0;jenkins-hbase20:43369] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-05-22 03:58:45,678 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-05-22 03:58:45,679 INFO [M:0;jenkins-hbase20:43369] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:43369 2023-05-22 03:58:45,681 DEBUG [M:0;jenkins-hbase20:43369] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,43369,1684727837208 already deleted, retry=false 2023-05-22 03:58:45,745 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): regionserver:43747-0x1017150ca2d0001, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:58:45,745 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): regionserver:43747-0x1017150ca2d0001, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:58:45,745 INFO [RS:0;jenkins-hbase20:43747] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,43747,1684727837286; zookeeper connection closed. 2023-05-22 03:58:45,746 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5446a336] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5446a336 2023-05-22 03:58:45,746 INFO [Listener at localhost.localdomain/33915] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-05-22 03:58:45,845 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:58:45,845 INFO [M:0;jenkins-hbase20:43369] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,43369,1684727837208; zookeeper connection closed. 2023-05-22 03:58:45,845 DEBUG [Listener at localhost.localdomain/33915-EventThread] zookeeper.ZKWatcher(600): master:43369-0x1017150ca2d0000, quorum=127.0.0.1:59657, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:58:45,848 WARN [Listener at localhost.localdomain/33915] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:58:45,857 INFO [Listener at localhost.localdomain/33915] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:58:45,972 WARN [BP-199956196-148.251.75.209-1684727836738 heartbeating to localhost.localdomain/127.0.0.1:45435] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:58:45,972 WARN [BP-199956196-148.251.75.209-1684727836738 heartbeating to localhost.localdomain/127.0.0.1:45435] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-199956196-148.251.75.209-1684727836738 (Datanode Uuid e53251e3-b749-4350-8e50-b4b41a9a4116) service to localhost.localdomain/127.0.0.1:45435 2023-05-22 03:58:45,973 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/cluster_13fb7289-2763-9201-0a7f-ce2633f64fd7/dfs/data/data3/current/BP-199956196-148.251.75.209-1684727836738] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:58:45,974 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/cluster_13fb7289-2763-9201-0a7f-ce2633f64fd7/dfs/data/data4/current/BP-199956196-148.251.75.209-1684727836738] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:58:45,975 WARN [Listener at localhost.localdomain/33915] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:58:45,979 INFO [Listener at localhost.localdomain/33915] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:58:46,037 WARN [BP-199956196-148.251.75.209-1684727836738 heartbeating to localhost.localdomain/127.0.0.1:45435] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-199956196-148.251.75.209-1684727836738 (Datanode Uuid 39c20281-e2ec-4825-9b9d-2c5aee149ecf) service to localhost.localdomain/127.0.0.1:45435 2023-05-22 03:58:46,038 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/cluster_13fb7289-2763-9201-0a7f-ce2633f64fd7/dfs/data/data1/current/BP-199956196-148.251.75.209-1684727836738] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:58:46,039 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/cluster_13fb7289-2763-9201-0a7f-ce2633f64fd7/dfs/data/data2/current/BP-199956196-148.251.75.209-1684727836738] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:58:46,107 INFO [Listener at localhost.localdomain/33915] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-05-22 03:58:46,231 INFO [Listener at localhost.localdomain/33915] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-05-22 03:58:46,263 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-05-22 03:58:46,272 INFO [Listener at localhost.localdomain/33915] hbase.ResourceChecker(175): after: regionserver.wal.TestLogRolling#testLogRolling Thread=107 (was 96) - Thread LEAK? -, OpenFileDescriptor=539 (was 499) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=60 (was 53) - SystemLoadAverage LEAK? -, ProcessCount=164 (was 165), AvailableMemoryMB=6204 (was 6600) 2023-05-22 03:58:46,280 INFO [Listener at localhost.localdomain/33915] hbase.ResourceChecker(147): before: regionserver.wal.TestLogRolling#testLogRollOnNothingWritten Thread=107, OpenFileDescriptor=539, MaxFileDescriptor=60000, SystemLoadAverage=60, ProcessCount=164, AvailableMemoryMB=6204 2023-05-22 03:58:46,280 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-05-22 03:58:46,281 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/hadoop.log.dir so I do NOT create it in target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e 2023-05-22 03:58:46,281 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/5d36a28a-50df-7dec-ea3b-291e7a3028e8/hadoop.tmp.dir so I do NOT create it in target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e 2023-05-22 03:58:46,281 INFO [Listener at localhost.localdomain/33915] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/cluster_99ddb2a9-5bdc-f491-1c94-ed3f0794c7f0, deleteOnExit=true 2023-05-22 03:58:46,281 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-05-22 03:58:46,281 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/test.cache.data in system properties and HBase conf 2023-05-22 03:58:46,281 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/hadoop.tmp.dir in system properties and HBase conf 2023-05-22 03:58:46,281 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/hadoop.log.dir in system properties and HBase conf 2023-05-22 03:58:46,281 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/mapreduce.cluster.local.dir in system properties and HBase conf 2023-05-22 03:58:46,281 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-05-22 03:58:46,282 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-05-22 03:58:46,282 DEBUG [Listener at localhost.localdomain/33915] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-05-22 03:58:46,282 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-05-22 03:58:46,282 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-05-22 03:58:46,282 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-05-22 03:58:46,282 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-05-22 03:58:46,282 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-05-22 03:58:46,283 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-05-22 03:58:46,283 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-05-22 03:58:46,283 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/dfs.journalnode.edits.dir in system properties and HBase conf 2023-05-22 03:58:46,283 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-05-22 03:58:46,283 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/nfs.dump.dir in system properties and HBase conf 2023-05-22 03:58:46,283 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/java.io.tmpdir in system properties and HBase conf 2023-05-22 03:58:46,283 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/dfs.journalnode.edits.dir in system properties and HBase conf 2023-05-22 03:58:46,283 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-05-22 03:58:46,284 INFO [Listener at localhost.localdomain/33915] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-05-22 03:58:46,285 WARN [Listener at localhost.localdomain/33915] blockmanagement.DatanodeManager(362): The given interval for marking stale datanode = 30000, which is larger than heartbeat expire interval 20000. 2023-05-22 03:58:46,286 WARN [Listener at localhost.localdomain/33915] conf.Configuration(1701): No unit for dfs.heartbeat.interval(1) assuming SECONDS 2023-05-22 03:58:46,287 WARN [Listener at localhost.localdomain/33915] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-05-22 03:58:46,309 WARN [Listener at localhost.localdomain/33915] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:58:46,311 INFO [Listener at localhost.localdomain/33915] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:58:46,315 INFO [Listener at localhost.localdomain/33915] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/java.io.tmpdir/Jetty_localhost_localdomain_36227_hdfs____1q9dam/webapp 2023-05-22 03:58:46,385 INFO [Listener at localhost.localdomain/33915] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:36227 2023-05-22 03:58:46,387 WARN [Listener at localhost.localdomain/33915] blockmanagement.DatanodeManager(362): The given interval for marking stale datanode = 30000, which is larger than heartbeat expire interval 20000. 2023-05-22 03:58:46,388 WARN [Listener at localhost.localdomain/33915] conf.Configuration(1701): No unit for dfs.heartbeat.interval(1) assuming SECONDS 2023-05-22 03:58:46,388 WARN [Listener at localhost.localdomain/33915] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-05-22 03:58:46,413 WARN [Listener at localhost.localdomain/34135] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:58:46,428 WARN [Listener at localhost.localdomain/34135] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:58:46,430 WARN [Listener at localhost.localdomain/34135] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:58:46,431 INFO [Listener at localhost.localdomain/34135] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:58:46,435 INFO [Listener at localhost.localdomain/34135] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/java.io.tmpdir/Jetty_localhost_45257_datanode____g1onmi/webapp 2023-05-22 03:58:46,504 INFO [Listener at localhost.localdomain/34135] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45257 2023-05-22 03:58:46,509 WARN [Listener at localhost.localdomain/40597] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:58:46,519 WARN [Listener at localhost.localdomain/40597] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-05-22 03:58:46,521 WARN [Listener at localhost.localdomain/40597] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-05-22 03:58:46,522 INFO [Listener at localhost.localdomain/40597] log.Slf4jLog(67): jetty-6.1.26 2023-05-22 03:58:46,525 INFO [Listener at localhost.localdomain/40597] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/java.io.tmpdir/Jetty_localhost_33665_datanode____.ufqutg/webapp 2023-05-22 03:58:46,592 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xda04f18cb57689f5: Processing first storage report for DS-6e904f26-8aa6-4b40-9f9a-293f2f81be6c from datanode 1174164b-ed90-4df5-8325-b2cfcad10c77 2023-05-22 03:58:46,592 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xda04f18cb57689f5: from storage DS-6e904f26-8aa6-4b40-9f9a-293f2f81be6c node DatanodeRegistration(127.0.0.1:39667, datanodeUuid=1174164b-ed90-4df5-8325-b2cfcad10c77, infoPort=32917, infoSecurePort=0, ipcPort=40597, storageInfo=lv=-57;cid=testClusterID;nsid=876482510;c=1684727926288), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:58:46,592 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xda04f18cb57689f5: Processing first storage report for DS-21cff078-6589-4857-95ad-4315ad92e9aa from datanode 1174164b-ed90-4df5-8325-b2cfcad10c77 2023-05-22 03:58:46,592 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xda04f18cb57689f5: from storage DS-21cff078-6589-4857-95ad-4315ad92e9aa node DatanodeRegistration(127.0.0.1:39667, datanodeUuid=1174164b-ed90-4df5-8325-b2cfcad10c77, infoPort=32917, infoSecurePort=0, ipcPort=40597, storageInfo=lv=-57;cid=testClusterID;nsid=876482510;c=1684727926288), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:58:46,601 INFO [Listener at localhost.localdomain/40597] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33665 2023-05-22 03:58:46,607 WARN [Listener at localhost.localdomain/34755] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-05-22 03:58:46,668 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2ed892799a7dfc70: Processing first storage report for DS-00efcfb9-a2e3-4674-b643-215264864647 from datanode 8b5c3207-7018-42ef-ae32-26a88aad3c18 2023-05-22 03:58:46,668 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2ed892799a7dfc70: from storage DS-00efcfb9-a2e3-4674-b643-215264864647 node DatanodeRegistration(127.0.0.1:35107, datanodeUuid=8b5c3207-7018-42ef-ae32-26a88aad3c18, infoPort=40043, infoSecurePort=0, ipcPort=34755, storageInfo=lv=-57;cid=testClusterID;nsid=876482510;c=1684727926288), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:58:46,669 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2ed892799a7dfc70: Processing first storage report for DS-57640c8a-9722-4bd1-b2b5-54e27a0b9dbf from datanode 8b5c3207-7018-42ef-ae32-26a88aad3c18 2023-05-22 03:58:46,669 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2ed892799a7dfc70: from storage DS-57640c8a-9722-4bd1-b2b5-54e27a0b9dbf node DatanodeRegistration(127.0.0.1:35107, datanodeUuid=8b5c3207-7018-42ef-ae32-26a88aad3c18, infoPort=40043, infoSecurePort=0, ipcPort=34755, storageInfo=lv=-57;cid=testClusterID;nsid=876482510;c=1684727926288), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-05-22 03:58:46,716 DEBUG [Listener at localhost.localdomain/34755] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e 2023-05-22 03:58:46,720 INFO [Listener at localhost.localdomain/34755] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/cluster_99ddb2a9-5bdc-f491-1c94-ed3f0794c7f0/zookeeper_0, clientPort=54526, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/cluster_99ddb2a9-5bdc-f491-1c94-ed3f0794c7f0/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/cluster_99ddb2a9-5bdc-f491-1c94-ed3f0794c7f0/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-05-22 03:58:46,722 INFO [Listener at localhost.localdomain/34755] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=54526 2023-05-22 03:58:46,722 INFO [Listener at localhost.localdomain/34755] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:58:46,724 INFO [Listener at localhost.localdomain/34755] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:58:46,742 INFO [Listener at localhost.localdomain/34755] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e with version=8 2023-05-22 03:58:46,742 INFO [Listener at localhost.localdomain/34755] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost.localdomain:36103/user/jenkins/test-data/9aa37ed3-5cff-516e-bca8-e5dfbcd75918/hbase-staging 2023-05-22 03:58:46,745 INFO [Listener at localhost.localdomain/34755] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-05-22 03:58:46,745 INFO [Listener at localhost.localdomain/34755] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:58:46,745 INFO [Listener at localhost.localdomain/34755] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-05-22 03:58:46,746 INFO [Listener at localhost.localdomain/34755] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-05-22 03:58:46,746 INFO [Listener at localhost.localdomain/34755] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:58:46,746 INFO [Listener at localhost.localdomain/34755] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-05-22 03:58:46,746 INFO [Listener at localhost.localdomain/34755] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-05-22 03:58:46,748 INFO [Listener at localhost.localdomain/34755] ipc.NettyRpcServer(120): Bind to /148.251.75.209:43371 2023-05-22 03:58:46,748 INFO [Listener at localhost.localdomain/34755] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:58:46,749 INFO [Listener at localhost.localdomain/34755] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:58:46,750 INFO [Listener at localhost.localdomain/34755] zookeeper.RecoverableZooKeeper(93): Process identifier=master:43371 connecting to ZooKeeper ensemble=127.0.0.1:54526 2023-05-22 03:58:46,754 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:433710x0, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-05-22 03:58:46,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:43371-0x101715227eb0000 connected 2023-05-22 03:58:46,764 DEBUG [Listener at localhost.localdomain/34755] zookeeper.ZKUtil(164): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:58:46,764 DEBUG [Listener at localhost.localdomain/34755] zookeeper.ZKUtil(164): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:58:46,765 DEBUG [Listener at localhost.localdomain/34755] zookeeper.ZKUtil(164): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-05-22 03:58:46,765 DEBUG [Listener at localhost.localdomain/34755] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43371 2023-05-22 03:58:46,765 DEBUG [Listener at localhost.localdomain/34755] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43371 2023-05-22 03:58:46,766 DEBUG [Listener at localhost.localdomain/34755] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43371 2023-05-22 03:58:46,766 DEBUG [Listener at localhost.localdomain/34755] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43371 2023-05-22 03:58:46,766 DEBUG [Listener at localhost.localdomain/34755] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43371 2023-05-22 03:58:46,767 INFO [Listener at localhost.localdomain/34755] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e, hbase.cluster.distributed=false 2023-05-22 03:58:46,780 INFO [Listener at localhost.localdomain/34755] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-05-22 03:58:46,780 INFO [Listener at localhost.localdomain/34755] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:58:46,780 INFO [Listener at localhost.localdomain/34755] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-05-22 03:58:46,780 INFO [Listener at localhost.localdomain/34755] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-05-22 03:58:46,780 INFO [Listener at localhost.localdomain/34755] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-05-22 03:58:46,780 INFO [Listener at localhost.localdomain/34755] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-05-22 03:58:46,781 INFO [Listener at localhost.localdomain/34755] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-05-22 03:58:46,782 INFO [Listener at localhost.localdomain/34755] ipc.NettyRpcServer(120): Bind to /148.251.75.209:34813 2023-05-22 03:58:46,782 INFO [Listener at localhost.localdomain/34755] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-05-22 03:58:46,785 DEBUG [Listener at localhost.localdomain/34755] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-05-22 03:58:46,786 INFO [Listener at localhost.localdomain/34755] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:58:46,786 INFO [Listener at localhost.localdomain/34755] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:58:46,787 INFO [Listener at localhost.localdomain/34755] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34813 connecting to ZooKeeper ensemble=127.0.0.1:54526 2023-05-22 03:58:46,791 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): regionserver:348130x0, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-05-22 03:58:46,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:34813-0x101715227eb0001 connected 2023-05-22 03:58:46,792 DEBUG [Listener at localhost.localdomain/34755] zookeeper.ZKUtil(164): regionserver:34813-0x101715227eb0001, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:58:46,793 DEBUG [Listener at localhost.localdomain/34755] zookeeper.ZKUtil(164): regionserver:34813-0x101715227eb0001, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:58:46,793 DEBUG [Listener at localhost.localdomain/34755] zookeeper.ZKUtil(164): regionserver:34813-0x101715227eb0001, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-05-22 03:58:46,793 DEBUG [Listener at localhost.localdomain/34755] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34813 2023-05-22 03:58:46,794 DEBUG [Listener at localhost.localdomain/34755] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34813 2023-05-22 03:58:46,794 DEBUG [Listener at localhost.localdomain/34755] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34813 2023-05-22 03:58:46,794 DEBUG [Listener at localhost.localdomain/34755] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34813 2023-05-22 03:58:46,794 DEBUG [Listener at localhost.localdomain/34755] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34813 2023-05-22 03:58:46,795 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,43371,1684727926744 2023-05-22 03:58:46,796 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-05-22 03:58:46,797 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,43371,1684727926744 2023-05-22 03:58:46,797 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-05-22 03:58:46,797 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:58:46,797 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): regionserver:34813-0x101715227eb0001, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-05-22 03:58:46,798 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-05-22 03:58:46,799 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,43371,1684727926744 from backup master directory 2023-05-22 03:58:46,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-05-22 03:58:46,800 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,43371,1684727926744 2023-05-22 03:58:46,800 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-05-22 03:58:46,800 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-05-22 03:58:46,800 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,43371,1684727926744 2023-05-22 03:58:46,817 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/hbase.id with ID: a9795081-cc6e-4408-8036-05aade95576a 2023-05-22 03:58:46,825 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:58:46,830 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:58:46,840 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x51da4c69 to 127.0.0.1:54526 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:58:46,844 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@436a1b15, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:58:46,844 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-05-22 03:58:46,845 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-05-22 03:58:46,845 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:58:46,846 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/MasterData/data/master/store-tmp 2023-05-22 03:58:46,853 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:58:46,853 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-05-22 03:58:46,853 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:58:46,853 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:58:46,853 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-05-22 03:58:46,853 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:58:46,853 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:58:46,853 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:58:46,854 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/MasterData/WALs/jenkins-hbase20.apache.org,43371,1684727926744 2023-05-22 03:58:46,856 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C43371%2C1684727926744, suffix=, logDir=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/MasterData/WALs/jenkins-hbase20.apache.org,43371,1684727926744, archiveDir=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/MasterData/oldWALs, maxLogs=10 2023-05-22 03:58:46,861 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/MasterData/WALs/jenkins-hbase20.apache.org,43371,1684727926744/jenkins-hbase20.apache.org%2C43371%2C1684727926744.1684727926856 2023-05-22 03:58:46,861 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39667,DS-6e904f26-8aa6-4b40-9f9a-293f2f81be6c,DISK], DatanodeInfoWithStorage[127.0.0.1:35107,DS-00efcfb9-a2e3-4674-b643-215264864647,DISK]] 2023-05-22 03:58:46,861 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:58:46,861 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:58:46,861 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:58:46,862 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:58:46,864 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:58:46,866 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-05-22 03:58:46,866 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-05-22 03:58:46,867 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:58:46,868 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:58:46,868 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:58:46,872 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-05-22 03:58:46,874 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:58:46,875 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=799775, jitterRate=0.01696762442588806}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:58:46,875 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:58:46,875 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-05-22 03:58:46,876 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-05-22 03:58:46,876 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-05-22 03:58:46,876 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-05-22 03:58:46,877 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-05-22 03:58:46,877 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-05-22 03:58:46,877 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-05-22 03:58:46,878 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-05-22 03:58:46,878 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-05-22 03:58:46,887 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-05-22 03:58:46,887 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-22 03:58:46,888 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-05-22 03:58:46,888 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-05-22 03:58:46,888 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-05-22 03:58:46,890 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:58:46,890 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-05-22 03:58:46,891 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-05-22 03:58:46,891 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-05-22 03:58:46,892 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-05-22 03:58:46,892 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): regionserver:34813-0x101715227eb0001, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-05-22 03:58:46,892 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:58:46,892 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,43371,1684727926744, sessionid=0x101715227eb0000, setting cluster-up flag (Was=false) 2023-05-22 03:58:46,895 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:58:46,897 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-05-22 03:58:46,898 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,43371,1684727926744 2023-05-22 03:58:46,899 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:58:46,901 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-05-22 03:58:46,902 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,43371,1684727926744 2023-05-22 03:58:46,903 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/.hbase-snapshot/.tmp 2023-05-22 03:58:46,905 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-05-22 03:58:46,905 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:58:46,906 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:58:46,906 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:58:46,906 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-05-22 03:58:46,906 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-05-22 03:58:46,906 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:58:46,906 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-05-22 03:58:46,906 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:58:46,911 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1684727956911 2023-05-22 03:58:46,911 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-05-22 03:58:46,911 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-05-22 03:58:46,911 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-05-22 03:58:46,911 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-05-22 03:58:46,911 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-05-22 03:58:46,911 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-05-22 03:58:46,911 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-05-22 03:58:46,912 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-05-22 03:58:46,912 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-05-22 03:58:46,912 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-05-22 03:58:46,912 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-05-22 03:58:46,912 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-05-22 03:58:46,912 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-05-22 03:58:46,912 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-05-22 03:58:46,912 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727926912,5,FailOnTimeoutGroup] 2023-05-22 03:58:46,912 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727926912,5,FailOnTimeoutGroup] 2023-05-22 03:58:46,913 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-05-22 03:58:46,913 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-05-22 03:58:46,913 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-05-22 03:58:46,913 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-05-22 03:58:46,913 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-05-22 03:58:46,924 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-05-22 03:58:46,925 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-05-22 03:58:46,925 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e 2023-05-22 03:58:46,935 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:58:46,938 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-05-22 03:58:46,939 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/info 2023-05-22 03:58:46,940 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-05-22 03:58:46,940 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:58:46,940 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-05-22 03:58:46,941 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:58:46,942 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-05-22 03:58:46,942 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:58:46,942 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-05-22 03:58:46,943 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/table 2023-05-22 03:58:46,944 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-05-22 03:58:46,944 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:58:46,945 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740 2023-05-22 03:58:46,945 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740 2023-05-22 03:58:46,946 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-05-22 03:58:46,947 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-05-22 03:58:46,949 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:58:46,949 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=864706, jitterRate=0.09953071177005768}}}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-05-22 03:58:46,949 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-05-22 03:58:46,949 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-05-22 03:58:46,949 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-05-22 03:58:46,949 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-05-22 03:58:46,949 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-05-22 03:58:46,949 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-05-22 03:58:46,950 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-05-22 03:58:46,950 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-05-22 03:58:46,950 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-05-22 03:58:46,950 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-05-22 03:58:46,950 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-05-22 03:58:46,952 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-05-22 03:58:46,953 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-05-22 03:58:46,996 INFO [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(951): ClusterId : a9795081-cc6e-4408-8036-05aade95576a 2023-05-22 03:58:46,997 DEBUG [RS:0;jenkins-hbase20:34813] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-05-22 03:58:46,999 DEBUG [RS:0;jenkins-hbase20:34813] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-05-22 03:58:46,999 DEBUG [RS:0;jenkins-hbase20:34813] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-05-22 03:58:47,001 DEBUG [RS:0;jenkins-hbase20:34813] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-05-22 03:58:47,002 DEBUG [RS:0;jenkins-hbase20:34813] zookeeper.ReadOnlyZKClient(139): Connect 0x238aeea6 to 127.0.0.1:54526 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:58:47,008 DEBUG [RS:0;jenkins-hbase20:34813] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d4d14a7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:58:47,008 DEBUG [RS:0;jenkins-hbase20:34813] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@297285d6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-05-22 03:58:47,018 DEBUG [RS:0;jenkins-hbase20:34813] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:34813 2023-05-22 03:58:47,018 INFO [RS:0;jenkins-hbase20:34813] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-05-22 03:58:47,018 INFO [RS:0;jenkins-hbase20:34813] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-05-22 03:58:47,019 DEBUG [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(1022): About to register with Master. 2023-05-22 03:58:47,019 INFO [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(2809): reportForDuty to master=jenkins-hbase20.apache.org,43371,1684727926744 with isa=jenkins-hbase20.apache.org/148.251.75.209:34813, startcode=1684727926780 2023-05-22 03:58:47,020 DEBUG [RS:0;jenkins-hbase20:34813] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-05-22 03:58:47,023 INFO [RS-EventLoopGroup-14-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43833, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.6 (auth:SIMPLE), service=RegionServerStatusService 2023-05-22 03:58:47,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43371] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,34813,1684727926780 2023-05-22 03:58:47,024 DEBUG [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e 2023-05-22 03:58:47,024 DEBUG [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:34135 2023-05-22 03:58:47,024 DEBUG [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-05-22 03:58:47,025 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:58:47,026 DEBUG [RS:0;jenkins-hbase20:34813] zookeeper.ZKUtil(162): regionserver:34813-0x101715227eb0001, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34813,1684727926780 2023-05-22 03:58:47,026 WARN [RS:0;jenkins-hbase20:34813] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-05-22 03:58:47,026 INFO [RS:0;jenkins-hbase20:34813] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:58:47,026 DEBUG [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(1946): logDir=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/WALs/jenkins-hbase20.apache.org,34813,1684727926780 2023-05-22 03:58:47,026 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,34813,1684727926780] 2023-05-22 03:58:47,030 DEBUG [RS:0;jenkins-hbase20:34813] zookeeper.ZKUtil(162): regionserver:34813-0x101715227eb0001, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34813,1684727926780 2023-05-22 03:58:47,030 DEBUG [RS:0;jenkins-hbase20:34813] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-05-22 03:58:47,031 INFO [RS:0;jenkins-hbase20:34813] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-05-22 03:58:47,031 INFO [RS:0;jenkins-hbase20:34813] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-05-22 03:58:47,032 INFO [RS:0;jenkins-hbase20:34813] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-05-22 03:58:47,032 INFO [RS:0;jenkins-hbase20:34813] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:58:47,032 INFO [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-05-22 03:58:47,033 INFO [RS:0;jenkins-hbase20:34813] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-05-22 03:58:47,033 DEBUG [RS:0;jenkins-hbase20:34813] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:58:47,033 DEBUG [RS:0;jenkins-hbase20:34813] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:58:47,033 DEBUG [RS:0;jenkins-hbase20:34813] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:58:47,033 DEBUG [RS:0;jenkins-hbase20:34813] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:58:47,033 DEBUG [RS:0;jenkins-hbase20:34813] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:58:47,033 DEBUG [RS:0;jenkins-hbase20:34813] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-05-22 03:58:47,033 DEBUG [RS:0;jenkins-hbase20:34813] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:58:47,033 DEBUG [RS:0;jenkins-hbase20:34813] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:58:47,033 DEBUG [RS:0;jenkins-hbase20:34813] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:58:47,033 DEBUG [RS:0;jenkins-hbase20:34813] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-05-22 03:58:47,034 INFO [RS:0;jenkins-hbase20:34813] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-05-22 03:58:47,034 INFO [RS:0;jenkins-hbase20:34813] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-05-22 03:58:47,034 INFO [RS:0;jenkins-hbase20:34813] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-05-22 03:58:47,043 INFO [RS:0;jenkins-hbase20:34813] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-05-22 03:58:47,043 INFO [RS:0;jenkins-hbase20:34813] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34813,1684727926780-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:58:47,052 INFO [RS:0;jenkins-hbase20:34813] regionserver.Replication(203): jenkins-hbase20.apache.org,34813,1684727926780 started 2023-05-22 03:58:47,052 INFO [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,34813,1684727926780, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:34813, sessionid=0x101715227eb0001 2023-05-22 03:58:47,052 DEBUG [RS:0;jenkins-hbase20:34813] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-05-22 03:58:47,052 DEBUG [RS:0;jenkins-hbase20:34813] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,34813,1684727926780 2023-05-22 03:58:47,052 DEBUG [RS:0;jenkins-hbase20:34813] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,34813,1684727926780' 2023-05-22 03:58:47,052 DEBUG [RS:0;jenkins-hbase20:34813] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-05-22 03:58:47,052 DEBUG [RS:0;jenkins-hbase20:34813] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-05-22 03:58:47,053 DEBUG [RS:0;jenkins-hbase20:34813] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-05-22 03:58:47,053 DEBUG [RS:0;jenkins-hbase20:34813] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-05-22 03:58:47,053 DEBUG [RS:0;jenkins-hbase20:34813] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,34813,1684727926780 2023-05-22 03:58:47,053 DEBUG [RS:0;jenkins-hbase20:34813] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,34813,1684727926780' 2023-05-22 03:58:47,053 DEBUG [RS:0;jenkins-hbase20:34813] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-05-22 03:58:47,053 DEBUG [RS:0;jenkins-hbase20:34813] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-05-22 03:58:47,054 DEBUG [RS:0;jenkins-hbase20:34813] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-05-22 03:58:47,054 INFO [RS:0;jenkins-hbase20:34813] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-05-22 03:58:47,054 INFO [RS:0;jenkins-hbase20:34813] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-05-22 03:58:47,103 DEBUG [jenkins-hbase20:43371] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-05-22 03:58:47,104 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,34813,1684727926780, state=OPENING 2023-05-22 03:58:47,105 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-05-22 03:58:47,107 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:58:47,108 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,34813,1684727926780}] 2023-05-22 03:58:47,108 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-05-22 03:58:47,157 INFO [RS:0;jenkins-hbase20:34813] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34813%2C1684727926780, suffix=, logDir=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/WALs/jenkins-hbase20.apache.org,34813,1684727926780, archiveDir=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/oldWALs, maxLogs=32 2023-05-22 03:58:47,169 INFO [RS:0;jenkins-hbase20:34813] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/WALs/jenkins-hbase20.apache.org,34813,1684727926780/jenkins-hbase20.apache.org%2C34813%2C1684727926780.1684727927158 2023-05-22 03:58:47,169 DEBUG [RS:0;jenkins-hbase20:34813] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35107,DS-00efcfb9-a2e3-4674-b643-215264864647,DISK], DatanodeInfoWithStorage[127.0.0.1:39667,DS-6e904f26-8aa6-4b40-9f9a-293f2f81be6c,DISK]] 2023-05-22 03:58:47,264 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,34813,1684727926780 2023-05-22 03:58:47,264 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-05-22 03:58:47,268 INFO [RS-EventLoopGroup-15-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48954, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-05-22 03:58:47,274 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-05-22 03:58:47,274 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:58:47,276 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34813%2C1684727926780.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/WALs/jenkins-hbase20.apache.org,34813,1684727926780, archiveDir=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/oldWALs, maxLogs=32 2023-05-22 03:58:47,282 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/WALs/jenkins-hbase20.apache.org,34813,1684727926780/jenkins-hbase20.apache.org%2C34813%2C1684727926780.meta.1684727927277.meta 2023-05-22 03:58:47,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39667,DS-6e904f26-8aa6-4b40-9f9a-293f2f81be6c,DISK], DatanodeInfoWithStorage[127.0.0.1:35107,DS-00efcfb9-a2e3-4674-b643-215264864647,DISK]] 2023-05-22 03:58:47,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:58:47,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-05-22 03:58:47,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-05-22 03:58:47,282 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-05-22 03:58:47,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-05-22 03:58:47,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:58:47,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-05-22 03:58:47,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-05-22 03:58:47,283 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-05-22 03:58:47,284 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/info 2023-05-22 03:58:47,284 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/info 2023-05-22 03:58:47,285 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-05-22 03:58:47,285 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:58:47,285 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-05-22 03:58:47,286 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:58:47,286 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/rep_barrier 2023-05-22 03:58:47,286 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-05-22 03:58:47,287 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:58:47,287 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-05-22 03:58:47,288 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/table 2023-05-22 03:58:47,288 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/table 2023-05-22 03:58:47,288 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-05-22 03:58:47,289 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:58:47,289 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740 2023-05-22 03:58:47,290 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740 2023-05-22 03:58:47,293 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-05-22 03:58:47,294 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-05-22 03:58:47,295 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=866013, jitterRate=0.10119301080703735}}}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-05-22 03:58:47,295 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-05-22 03:58:47,298 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1684727927264 2023-05-22 03:58:47,302 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for hbase:meta,,1.1588230740 2023-05-22 03:58:47,303 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-05-22 03:58:47,304 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,34813,1684727926780, state=OPEN 2023-05-22 03:58:47,306 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-05-22 03:58:47,306 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-05-22 03:58:47,309 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-05-22 03:58:47,309 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,34813,1684727926780 in 198 msec 2023-05-22 03:58:47,312 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-05-22 03:58:47,312 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 359 msec 2023-05-22 03:58:47,315 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 410 msec 2023-05-22 03:58:47,316 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1684727927316, completionTime=-1 2023-05-22 03:58:47,316 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-05-22 03:58:47,316 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-05-22 03:58:47,320 DEBUG [hconnection-0x19040ad6-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-05-22 03:58:47,323 INFO [RS-EventLoopGroup-15-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48964, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-05-22 03:58:47,326 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-05-22 03:58:47,326 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1684727987326 2023-05-22 03:58:47,326 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1684728047326 2023-05-22 03:58:47,326 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 10 msec 2023-05-22 03:58:47,336 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43371,1684727926744-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-05-22 03:58:47,336 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43371,1684727926744-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:58:47,336 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43371,1684727926744-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:58:47,336 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:43371, period=300000, unit=MILLISECONDS is enabled. 2023-05-22 03:58:47,336 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-05-22 03:58:47,336 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-05-22 03:58:47,336 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-05-22 03:58:47,338 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-05-22 03:58:47,338 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-05-22 03:58:47,341 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-05-22 03:58:47,343 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-05-22 03:58:47,345 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/.tmp/data/hbase/namespace/e833237b9c02e621aa4c213befad542f 2023-05-22 03:58:47,346 DEBUG [HFileArchiver-11] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/.tmp/data/hbase/namespace/e833237b9c02e621aa4c213befad542f empty. 2023-05-22 03:58:47,347 DEBUG [HFileArchiver-11] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/.tmp/data/hbase/namespace/e833237b9c02e621aa4c213befad542f 2023-05-22 03:58:47,347 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-05-22 03:58:47,359 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-05-22 03:58:47,360 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => e833237b9c02e621aa4c213befad542f, NAME => 'hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/.tmp 2023-05-22 03:58:47,367 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:58:47,367 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing e833237b9c02e621aa4c213befad542f, disabling compactions & flushes 2023-05-22 03:58:47,367 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f. 2023-05-22 03:58:47,367 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f. 2023-05-22 03:58:47,368 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f. after waiting 0 ms 2023-05-22 03:58:47,368 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f. 2023-05-22 03:58:47,368 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f. 2023-05-22 03:58:47,368 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for e833237b9c02e621aa4c213befad542f: 2023-05-22 03:58:47,370 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-05-22 03:58:47,372 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727927371"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1684727927371"}]},"ts":"1684727927371"} 2023-05-22 03:58:47,374 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-05-22 03:58:47,375 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-05-22 03:58:47,375 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727927375"}]},"ts":"1684727927375"} 2023-05-22 03:58:47,376 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-05-22 03:58:47,381 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e833237b9c02e621aa4c213befad542f, ASSIGN}] 2023-05-22 03:58:47,383 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e833237b9c02e621aa4c213befad542f, ASSIGN 2023-05-22 03:58:47,384 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=e833237b9c02e621aa4c213befad542f, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34813,1684727926780; forceNewPlan=false, retain=false 2023-05-22 03:58:47,536 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e833237b9c02e621aa4c213befad542f, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34813,1684727926780 2023-05-22 03:58:47,537 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727927536"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1684727927536"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1684727927536"}]},"ts":"1684727927536"} 2023-05-22 03:58:47,541 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure e833237b9c02e621aa4c213befad542f, server=jenkins-hbase20.apache.org,34813,1684727926780}] 2023-05-22 03:58:47,702 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f. 2023-05-22 03:58:47,702 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e833237b9c02e621aa4c213befad542f, NAME => 'hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f.', STARTKEY => '', ENDKEY => ''} 2023-05-22 03:58:47,702 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace e833237b9c02e621aa4c213befad542f 2023-05-22 03:58:47,702 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-05-22 03:58:47,703 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for e833237b9c02e621aa4c213befad542f 2023-05-22 03:58:47,703 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for e833237b9c02e621aa4c213befad542f 2023-05-22 03:58:47,705 INFO [StoreOpener-e833237b9c02e621aa4c213befad542f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region e833237b9c02e621aa4c213befad542f 2023-05-22 03:58:47,706 DEBUG [StoreOpener-e833237b9c02e621aa4c213befad542f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/namespace/e833237b9c02e621aa4c213befad542f/info 2023-05-22 03:58:47,707 DEBUG [StoreOpener-e833237b9c02e621aa4c213befad542f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/namespace/e833237b9c02e621aa4c213befad542f/info 2023-05-22 03:58:47,707 INFO [StoreOpener-e833237b9c02e621aa4c213befad542f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e833237b9c02e621aa4c213befad542f columnFamilyName info 2023-05-22 03:58:47,707 INFO [StoreOpener-e833237b9c02e621aa4c213befad542f-1] regionserver.HStore(310): Store=e833237b9c02e621aa4c213befad542f/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-05-22 03:58:47,708 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/namespace/e833237b9c02e621aa4c213befad542f 2023-05-22 03:58:47,708 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/namespace/e833237b9c02e621aa4c213befad542f 2023-05-22 03:58:47,711 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for e833237b9c02e621aa4c213befad542f 2023-05-22 03:58:47,712 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/namespace/e833237b9c02e621aa4c213befad542f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-05-22 03:58:47,712 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened e833237b9c02e621aa4c213befad542f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=16384, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=714889, jitterRate=-0.0909728854894638}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-05-22 03:58:47,712 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for e833237b9c02e621aa4c213befad542f: 2023-05-22 03:58:47,714 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2334): Post open deploy tasks for hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f., pid=6, masterSystemTime=1684727927696 2023-05-22 03:58:47,716 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2361): Finished post open deploy task for hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f. 2023-05-22 03:58:47,716 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f. 2023-05-22 03:58:47,716 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e833237b9c02e621aa4c213befad542f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34813,1684727926780 2023-05-22 03:58:47,717 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1684727927716"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1684727927716"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1684727927716"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1684727927716"}]},"ts":"1684727927716"} 2023-05-22 03:58:47,719 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-05-22 03:58:47,719 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure e833237b9c02e621aa4c213befad542f, server=jenkins-hbase20.apache.org,34813,1684727926780 in 178 msec 2023-05-22 03:58:47,721 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-05-22 03:58:47,721 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=e833237b9c02e621aa4c213befad542f, ASSIGN in 339 msec 2023-05-22 03:58:47,722 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-05-22 03:58:47,722 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1684727927722"}]},"ts":"1684727927722"} 2023-05-22 03:58:47,724 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-05-22 03:58:47,725 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-05-22 03:58:47,726 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 388 msec 2023-05-22 03:58:47,740 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-05-22 03:58:47,747 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:58:47,747 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:58:47,750 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-05-22 03:58:47,758 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:58:47,761 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 10 msec 2023-05-22 03:58:47,772 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-05-22 03:58:47,778 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-05-22 03:58:47,782 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 10 msec 2023-05-22 03:58:47,785 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-05-22 03:58:47,787 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-05-22 03:58:47,787 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 0.987sec 2023-05-22 03:58:47,787 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-05-22 03:58:47,787 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-05-22 03:58:47,787 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-05-22 03:58:47,787 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43371,1684727926744-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-05-22 03:58:47,787 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43371,1684727926744-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-05-22 03:58:47,789 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-05-22 03:58:47,797 DEBUG [Listener at localhost.localdomain/34755] zookeeper.ReadOnlyZKClient(139): Connect 0x68d331b5 to 127.0.0.1:54526 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-05-22 03:58:47,805 DEBUG [Listener at localhost.localdomain/34755] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b3415fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-05-22 03:58:47,807 DEBUG [hconnection-0x7e764197-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-05-22 03:58:47,809 INFO [RS-EventLoopGroup-15-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48972, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-05-22 03:58:47,810 INFO [Listener at localhost.localdomain/34755] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,43371,1684727926744 2023-05-22 03:58:47,810 INFO [Listener at localhost.localdomain/34755] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-05-22 03:58:47,816 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-05-22 03:58:47,816 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:58:47,816 INFO [Listener at localhost.localdomain/34755] master.MasterRpcServices(492): Client=null/null set balanceSwitch=false 2023-05-22 03:58:47,817 INFO [Listener at localhost.localdomain/34755] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2023-05-22 03:58:47,819 INFO [Listener at localhost.localdomain/34755] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=test.com%2C8080%2C1, suffix=, logDir=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/WALs/test.com,8080,1, archiveDir=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/oldWALs, maxLogs=32 2023-05-22 03:58:47,831 INFO [Listener at localhost.localdomain/34755] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/WALs/test.com,8080,1/test.com%2C8080%2C1.1684727927819 2023-05-22 03:58:47,831 DEBUG [Listener at localhost.localdomain/34755] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35107,DS-00efcfb9-a2e3-4674-b643-215264864647,DISK], DatanodeInfoWithStorage[127.0.0.1:39667,DS-6e904f26-8aa6-4b40-9f9a-293f2f81be6c,DISK]] 2023-05-22 03:58:47,840 INFO [Listener at localhost.localdomain/34755] wal.AbstractFSWAL(802): Rolled WAL /user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/WALs/test.com,8080,1/test.com%2C8080%2C1.1684727927819 with entries=0, filesize=83 B; new WAL /user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/WALs/test.com,8080,1/test.com%2C8080%2C1.1684727927831 2023-05-22 03:58:47,842 DEBUG [Listener at localhost.localdomain/34755] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39667,DS-6e904f26-8aa6-4b40-9f9a-293f2f81be6c,DISK], DatanodeInfoWithStorage[127.0.0.1:35107,DS-00efcfb9-a2e3-4674-b643-215264864647,DISK]] 2023-05-22 03:58:47,842 DEBUG [Listener at localhost.localdomain/34755] wal.AbstractFSWAL(716): hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/WALs/test.com,8080,1/test.com%2C8080%2C1.1684727927819 is not closed yet, will try archiving it next time 2023-05-22 03:58:47,843 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/WALs/test.com,8080,1 2023-05-22 03:58:47,854 INFO [WAL-Archive-0] wal.AbstractFSWAL(783): Archiving hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/WALs/test.com,8080,1/test.com%2C8080%2C1.1684727927819 to hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/oldWALs/test.com%2C8080%2C1.1684727927819 2023-05-22 03:58:48,262 DEBUG [Listener at localhost.localdomain/34755] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/oldWALs 2023-05-22 03:58:48,262 INFO [Listener at localhost.localdomain/34755] wal.AbstractFSWAL(1031): Closed WAL: FSHLog test.com%2C8080%2C1:(num 1684727927831) 2023-05-22 03:58:48,262 INFO [Listener at localhost.localdomain/34755] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-05-22 03:58:48,262 DEBUG [Listener at localhost.localdomain/34755] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68d331b5 to 127.0.0.1:54526 2023-05-22 03:58:48,262 DEBUG [Listener at localhost.localdomain/34755] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:58:48,263 DEBUG [Listener at localhost.localdomain/34755] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-05-22 03:58:48,263 DEBUG [Listener at localhost.localdomain/34755] util.JVMClusterUtil(257): Found active master hash=237119384, stopped=false 2023-05-22 03:58:48,264 INFO [Listener at localhost.localdomain/34755] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,43371,1684727926744 2023-05-22 03:58:48,265 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): regionserver:34813-0x101715227eb0001, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-05-22 03:58:48,265 INFO [Listener at localhost.localdomain/34755] procedure2.ProcedureExecutor(629): Stopping 2023-05-22 03:58:48,265 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-05-22 03:58:48,266 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:58:48,266 DEBUG [Listener at localhost.localdomain/34755] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51da4c69 to 127.0.0.1:54526 2023-05-22 03:58:48,266 DEBUG [Listener at localhost.localdomain/34755] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:58:48,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34813-0x101715227eb0001, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:58:48,267 INFO [Listener at localhost.localdomain/34755] regionserver.HRegionServer(2295): ***** STOPPING region server 'jenkins-hbase20.apache.org,34813,1684727926780' ***** 2023-05-22 03:58:48,267 INFO [Listener at localhost.localdomain/34755] regionserver.HRegionServer(2309): STOPPED: Shutdown requested 2023-05-22 03:58:48,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-05-22 03:58:48,267 INFO [RS:0;jenkins-hbase20:34813] regionserver.HeapMemoryManager(220): Stopping 2023-05-22 03:58:48,267 INFO [RS:0;jenkins-hbase20:34813] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-05-22 03:58:48,267 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-05-22 03:58:48,267 INFO [RS:0;jenkins-hbase20:34813] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-05-22 03:58:48,268 INFO [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(3303): Received CLOSE for e833237b9c02e621aa4c213befad542f 2023-05-22 03:58:48,268 INFO [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,34813,1684727926780 2023-05-22 03:58:48,268 DEBUG [RS:0;jenkins-hbase20:34813] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x238aeea6 to 127.0.0.1:54526 2023-05-22 03:58:48,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing e833237b9c02e621aa4c213befad542f, disabling compactions & flushes 2023-05-22 03:58:48,269 DEBUG [RS:0;jenkins-hbase20:34813] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:58:48,269 INFO [RS:0;jenkins-hbase20:34813] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-05-22 03:58:48,269 INFO [RS:0;jenkins-hbase20:34813] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-05-22 03:58:48,269 INFO [RS:0;jenkins-hbase20:34813] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-05-22 03:58:48,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f. 2023-05-22 03:58:48,269 INFO [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(3303): Received CLOSE for 1588230740 2023-05-22 03:58:48,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f. 2023-05-22 03:58:48,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f. after waiting 0 ms 2023-05-22 03:58:48,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f. 2023-05-22 03:58:48,270 INFO [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-05-22 03:58:48,270 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing e833237b9c02e621aa4c213befad542f 1/1 column families, dataSize=78 B heapSize=488 B 2023-05-22 03:58:48,271 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-05-22 03:58:48,270 DEBUG [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, e833237b9c02e621aa4c213befad542f=hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f.} 2023-05-22 03:58:48,271 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-05-22 03:58:48,271 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-05-22 03:58:48,271 DEBUG [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(1504): Waiting on 1588230740, e833237b9c02e621aa4c213befad542f 2023-05-22 03:58:48,271 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-05-22 03:58:48,271 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-05-22 03:58:48,271 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.26 KB heapSize=2.89 KB 2023-05-22 03:58:48,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/namespace/e833237b9c02e621aa4c213befad542f/.tmp/info/9cd5cafe5bb9480b9af7721b00d78e2d 2023-05-22 03:58:48,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/namespace/e833237b9c02e621aa4c213befad542f/.tmp/info/9cd5cafe5bb9480b9af7721b00d78e2d as hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/namespace/e833237b9c02e621aa4c213befad542f/info/9cd5cafe5bb9480b9af7721b00d78e2d 2023-05-22 03:58:48,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/namespace/e833237b9c02e621aa4c213befad542f/info/9cd5cafe5bb9480b9af7721b00d78e2d, entries=2, sequenceid=6, filesize=4.8 K 2023-05-22 03:58:48,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for e833237b9c02e621aa4c213befad542f in 27ms, sequenceid=6, compaction requested=false 2023-05-22 03:58:48,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-05-22 03:58:48,303 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/namespace/e833237b9c02e621aa4c213befad542f/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-05-22 03:58:48,304 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f. 2023-05-22 03:58:48,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for e833237b9c02e621aa4c213befad542f: 2023-05-22 03:58:48,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1684727927336.e833237b9c02e621aa4c213befad542f. 2023-05-22 03:58:48,471 DEBUG [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-05-22 03:58:48,672 DEBUG [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-05-22 03:58:48,689 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.17 KB at sequenceid=9 (bloomFilter=false), to=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/.tmp/info/b8d1cd87c1ef433eb25326d53347937e 2023-05-22 03:58:48,710 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=94 B at sequenceid=9 (bloomFilter=false), to=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/.tmp/table/821ae9a92e0a4241917180d39a09dc7d 2023-05-22 03:58:48,716 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/.tmp/info/b8d1cd87c1ef433eb25326d53347937e as hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/info/b8d1cd87c1ef433eb25326d53347937e 2023-05-22 03:58:48,721 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/info/b8d1cd87c1ef433eb25326d53347937e, entries=10, sequenceid=9, filesize=5.9 K 2023-05-22 03:58:48,722 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/.tmp/table/821ae9a92e0a4241917180d39a09dc7d as hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/table/821ae9a92e0a4241917180d39a09dc7d 2023-05-22 03:58:48,728 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/table/821ae9a92e0a4241917180d39a09dc7d, entries=2, sequenceid=9, filesize=4.7 K 2023-05-22 03:58:48,729 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.26 KB/1292, heapSize ~2.61 KB/2672, currentSize=0 B/0 for 1588230740 in 458ms, sequenceid=9, compaction requested=false 2023-05-22 03:58:48,729 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-05-22 03:58:48,737 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/data/hbase/meta/1588230740/recovered.edits/12.seqid, newMaxSeqId=12, maxSeqId=1 2023-05-22 03:58:48,737 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-05-22 03:58:48,737 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-05-22 03:58:48,737 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-05-22 03:58:48,737 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-05-22 03:58:48,872 INFO [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,34813,1684727926780; all regions closed. 2023-05-22 03:58:48,873 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/WALs/jenkins-hbase20.apache.org,34813,1684727926780 2023-05-22 03:58:48,881 DEBUG [RS:0;jenkins-hbase20:34813] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/oldWALs 2023-05-22 03:58:48,881 INFO [RS:0;jenkins-hbase20:34813] wal.AbstractFSWAL(1031): Closed WAL: FSHLog jenkins-hbase20.apache.org%2C34813%2C1684727926780.meta:.meta(num 1684727927277) 2023-05-22 03:58:48,882 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/WALs/jenkins-hbase20.apache.org,34813,1684727926780 2023-05-22 03:58:48,890 DEBUG [RS:0;jenkins-hbase20:34813] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/oldWALs 2023-05-22 03:58:48,890 INFO [RS:0;jenkins-hbase20:34813] wal.AbstractFSWAL(1031): Closed WAL: FSHLog jenkins-hbase20.apache.org%2C34813%2C1684727926780:(num 1684727927158) 2023-05-22 03:58:48,890 DEBUG [RS:0;jenkins-hbase20:34813] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:58:48,890 INFO [RS:0;jenkins-hbase20:34813] regionserver.LeaseManager(133): Closed leases 2023-05-22 03:58:48,890 INFO [RS:0;jenkins-hbase20:34813] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-05-22 03:58:48,891 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-05-22 03:58:48,892 INFO [RS:0;jenkins-hbase20:34813] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:34813 2023-05-22 03:58:48,896 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): regionserver:34813-0x101715227eb0001, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,34813,1684727926780 2023-05-22 03:58:48,896 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:58:48,896 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): regionserver:34813-0x101715227eb0001, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-05-22 03:58:48,896 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,34813,1684727926780] 2023-05-22 03:58:48,896 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,34813,1684727926780; numProcessing=1 2023-05-22 03:58:48,897 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,34813,1684727926780 already deleted, retry=false 2023-05-22 03:58:48,897 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,34813,1684727926780 expired; onlineServers=0 2023-05-22 03:58:48,897 INFO [RegionServerTracker-0] regionserver.HRegionServer(2295): ***** STOPPING region server 'jenkins-hbase20.apache.org,43371,1684727926744' ***** 2023-05-22 03:58:48,897 INFO [RegionServerTracker-0] regionserver.HRegionServer(2309): STOPPED: Cluster shutdown set; onlineServer=0 2023-05-22 03:58:48,898 DEBUG [M:0;jenkins-hbase20:43371] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a1da284, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-05-22 03:58:48,898 INFO [M:0;jenkins-hbase20:43371] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,43371,1684727926744 2023-05-22 03:58:48,898 INFO [M:0;jenkins-hbase20:43371] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,43371,1684727926744; all regions closed. 2023-05-22 03:58:48,898 DEBUG [M:0;jenkins-hbase20:43371] ipc.AbstractRpcClient(494): Stopping rpc client 2023-05-22 03:58:48,898 DEBUG [M:0;jenkins-hbase20:43371] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-05-22 03:58:48,898 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-05-22 03:58:48,899 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727926912] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1684727926912,5,FailOnTimeoutGroup] 2023-05-22 03:58:48,899 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727926912] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1684727926912,5,FailOnTimeoutGroup] 2023-05-22 03:58:48,898 DEBUG [M:0;jenkins-hbase20:43371] cleaner.HFileCleaner(317): Stopping file delete threads 2023-05-22 03:58:48,900 INFO [M:0;jenkins-hbase20:43371] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-05-22 03:58:48,901 INFO [M:0;jenkins-hbase20:43371] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-05-22 03:58:48,901 INFO [M:0;jenkins-hbase20:43371] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-05-22 03:58:48,901 DEBUG [M:0;jenkins-hbase20:43371] master.HMaster(1512): Stopping service threads 2023-05-22 03:58:48,901 INFO [M:0;jenkins-hbase20:43371] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-05-22 03:58:48,902 ERROR [M:0;jenkins-hbase20:43371] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] 2023-05-22 03:58:48,902 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-05-22 03:58:48,902 INFO [M:0;jenkins-hbase20:43371] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-05-22 03:58:48,902 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-05-22 03:58:48,902 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-05-22 03:58:48,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-05-22 03:58:48,903 DEBUG [M:0;jenkins-hbase20:43371] zookeeper.ZKUtil(398): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-05-22 03:58:48,903 WARN [M:0;jenkins-hbase20:43371] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-05-22 03:58:48,903 INFO [M:0;jenkins-hbase20:43371] assignment.AssignmentManager(315): Stopping assignment manager 2023-05-22 03:58:48,903 INFO [M:0;jenkins-hbase20:43371] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-05-22 03:58:48,904 DEBUG [M:0;jenkins-hbase20:43371] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-05-22 03:58:48,904 INFO [M:0;jenkins-hbase20:43371] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:58:48,904 DEBUG [M:0;jenkins-hbase20:43371] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:58:48,904 DEBUG [M:0;jenkins-hbase20:43371] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-05-22 03:58:48,904 DEBUG [M:0;jenkins-hbase20:43371] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:58:48,904 INFO [M:0;jenkins-hbase20:43371] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=24.09 KB heapSize=29.59 KB 2023-05-22 03:58:48,918 INFO [M:0;jenkins-hbase20:43371] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.09 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/9c3d9c698e7b4355a3e4a8be877bf660 2023-05-22 03:58:48,922 DEBUG [M:0;jenkins-hbase20:43371] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/9c3d9c698e7b4355a3e4a8be877bf660 as hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/9c3d9c698e7b4355a3e4a8be877bf660 2023-05-22 03:58:48,926 INFO [M:0;jenkins-hbase20:43371] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34135/user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/9c3d9c698e7b4355a3e4a8be877bf660, entries=8, sequenceid=66, filesize=6.3 K 2023-05-22 03:58:48,927 INFO [M:0;jenkins-hbase20:43371] regionserver.HRegion(2948): Finished flush of dataSize ~24.09 KB/24669, heapSize ~29.57 KB/30280, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 23ms, sequenceid=66, compaction requested=false 2023-05-22 03:58:48,928 INFO [M:0;jenkins-hbase20:43371] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-05-22 03:58:48,928 DEBUG [M:0;jenkins-hbase20:43371] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-05-22 03:58:48,928 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /user/jenkins/test-data/af41400c-61a5-57f8-1213-65b01457a70e/MasterData/WALs/jenkins-hbase20.apache.org,43371,1684727926744 2023-05-22 03:58:48,931 INFO [M:0;jenkins-hbase20:43371] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-05-22 03:58:48,931 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-05-22 03:58:48,932 INFO [M:0;jenkins-hbase20:43371] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:43371 2023-05-22 03:58:48,934 DEBUG [M:0;jenkins-hbase20:43371] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,43371,1684727926744 already deleted, retry=false 2023-05-22 03:58:48,997 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): regionserver:34813-0x101715227eb0001, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:58:48,997 INFO [RS:0;jenkins-hbase20:34813] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,34813,1684727926780; zookeeper connection closed. 2023-05-22 03:58:48,997 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): regionserver:34813-0x101715227eb0001, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:58:48,998 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@57f66f6f] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@57f66f6f 2023-05-22 03:58:48,998 INFO [Listener at localhost.localdomain/34755] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-05-22 03:58:49,035 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-05-22 03:58:49,097 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:58:49,097 INFO [M:0;jenkins-hbase20:43371] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,43371,1684727926744; zookeeper connection closed. 2023-05-22 03:58:49,097 DEBUG [Listener at localhost.localdomain/34755-EventThread] zookeeper.ZKWatcher(600): master:43371-0x101715227eb0000, quorum=127.0.0.1:54526, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-05-22 03:58:49,100 WARN [Listener at localhost.localdomain/34755] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:58:49,108 INFO [Listener at localhost.localdomain/34755] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:58:49,214 WARN [BP-401766558-148.251.75.209-1684727926288 heartbeating to localhost.localdomain/127.0.0.1:34135] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:58:49,214 WARN [BP-401766558-148.251.75.209-1684727926288 heartbeating to localhost.localdomain/127.0.0.1:34135] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-401766558-148.251.75.209-1684727926288 (Datanode Uuid 8b5c3207-7018-42ef-ae32-26a88aad3c18) service to localhost.localdomain/127.0.0.1:34135 2023-05-22 03:58:49,215 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/cluster_99ddb2a9-5bdc-f491-1c94-ed3f0794c7f0/dfs/data/data3/current/BP-401766558-148.251.75.209-1684727926288] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:58:49,216 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/cluster_99ddb2a9-5bdc-f491-1c94-ed3f0794c7f0/dfs/data/data4/current/BP-401766558-148.251.75.209-1684727926288] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:58:49,216 WARN [Listener at localhost.localdomain/34755] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-05-22 03:58:49,220 INFO [Listener at localhost.localdomain/34755] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-05-22 03:58:49,324 WARN [BP-401766558-148.251.75.209-1684727926288 heartbeating to localhost.localdomain/127.0.0.1:34135] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-05-22 03:58:49,324 WARN [BP-401766558-148.251.75.209-1684727926288 heartbeating to localhost.localdomain/127.0.0.1:34135] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-401766558-148.251.75.209-1684727926288 (Datanode Uuid 1174164b-ed90-4df5-8325-b2cfcad10c77) service to localhost.localdomain/127.0.0.1:34135 2023-05-22 03:58:49,326 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/cluster_99ddb2a9-5bdc-f491-1c94-ed3f0794c7f0/dfs/data/data1/current/BP-401766558-148.251.75.209-1684727926288] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:58:49,326 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bdf5b2e2-6549-f757-239d-484bce8e0f9e/cluster_99ddb2a9-5bdc-f491-1c94-ed3f0794c7f0/dfs/data/data2/current/BP-401766558-148.251.75.209-1684727926288] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-05-22 03:58:49,340 INFO [Listener at localhost.localdomain/34755] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-05-22 03:58:49,457 INFO [Listener at localhost.localdomain/34755] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-05-22 03:58:49,467 INFO [Listener at localhost.localdomain/34755] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-05-22 03:58:49,477 INFO [Listener at localhost.localdomain/34755] hbase.ResourceChecker(175): after: regionserver.wal.TestLogRolling#testLogRollOnNothingWritten Thread=130 (was 107) - Thread LEAK? -, OpenFileDescriptor=563 (was 539) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=71 (was 60) - SystemLoadAverage LEAK? -, ProcessCount=164 (was 164), AvailableMemoryMB=6196 (was 6204)