2023-07-12 15:04:03,325 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures timeout: 13 mins 2023-07-12 15:04:03,544 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3 2023-07-12 15:04:03,577 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.procedure.TestMasterFailoverWithProcedures#testDisableTableWithFailover Thread=10, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=312, ProcessCount=161, AvailableMemoryMB=8756 2023-07-12 15:04:03,585 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=2, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-12 15:04:03,585 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/cluster_b99044a4-4030-b979-45d0-f7a87f1007da, deleteOnExit=true 2023-07-12 15:04:03,586 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-12 15:04:03,586 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/test.cache.data in system properties and HBase conf 2023-07-12 15:04:03,587 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 15:04:03,587 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/hadoop.log.dir in system properties and HBase conf 2023-07-12 15:04:03,588 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 15:04:03,588 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 15:04:03,588 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 15:04:03,713 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-12 15:04:04,089 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 15:04:04,093 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 15:04:04,093 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 15:04:04,093 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 15:04:04,094 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 15:04:04,094 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 15:04:04,094 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 15:04:04,094 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 15:04:04,095 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 15:04:04,095 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 15:04:04,095 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/nfs.dump.dir in system properties and HBase conf 2023-07-12 15:04:04,095 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/java.io.tmpdir in system properties and HBase conf 2023-07-12 15:04:04,096 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 15:04:04,096 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 15:04:04,096 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 15:04:04,558 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 15:04:04,562 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 15:04:04,876 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-12 15:04:05,028 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-12 15:04:05,042 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 15:04:05,080 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 15:04:05,145 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/java.io.tmpdir/Jetty_localhost_40387_hdfs____4a22ao/webapp 2023-07-12 15:04:05,287 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40387 2023-07-12 15:04:05,298 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 15:04:05,298 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 15:04:05,713 WARN [Listener at localhost/46679] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 15:04:05,778 WARN [Listener at localhost/46679] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 15:04:05,796 WARN [Listener at localhost/46679] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 15:04:05,801 INFO [Listener at localhost/46679] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 15:04:05,806 INFO [Listener at localhost/46679] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/java.io.tmpdir/Jetty_localhost_33075_datanode____op911x/webapp 2023-07-12 15:04:05,907 INFO [Listener at localhost/46679] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33075 2023-07-12 15:04:06,225 WARN [Listener at localhost/34167] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 15:04:06,660 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x9eceb21fc042b4a4: Processing first storage report for DS-10ff2eee-c043-4847-b4da-8d4bdc248f86 from datanode 761e383d-70b7-4b64-9aed-989e1d50a70b 2023-07-12 15:04:06,662 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x9eceb21fc042b4a4: from storage DS-10ff2eee-c043-4847-b4da-8d4bdc248f86 node DatanodeRegistration(127.0.0.1:45305, datanodeUuid=761e383d-70b7-4b64-9aed-989e1d50a70b, infoPort=33071, infoSecurePort=0, ipcPort=34167, storageInfo=lv=-57;cid=testClusterID;nsid=1161452585;c=1689174244642), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-12 15:04:06,662 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x9eceb21fc042b4a4: Processing first storage report for DS-26052ae4-a201-4789-8adf-5126e0e4c820 from datanode 761e383d-70b7-4b64-9aed-989e1d50a70b 2023-07-12 15:04:06,662 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x9eceb21fc042b4a4: from storage DS-26052ae4-a201-4789-8adf-5126e0e4c820 node DatanodeRegistration(127.0.0.1:45305, datanodeUuid=761e383d-70b7-4b64-9aed-989e1d50a70b, infoPort=33071, infoSecurePort=0, ipcPort=34167, storageInfo=lv=-57;cid=testClusterID;nsid=1161452585;c=1689174244642), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 15:04:06,743 DEBUG [Listener at localhost/34167] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3 2023-07-12 15:04:06,813 INFO [Listener at localhost/34167] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/cluster_b99044a4-4030-b979-45d0-f7a87f1007da/zookeeper_0, clientPort=61057, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/cluster_b99044a4-4030-b979-45d0-f7a87f1007da/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/cluster_b99044a4-4030-b979-45d0-f7a87f1007da/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 15:04:06,828 INFO [Listener at localhost/34167] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=61057 2023-07-12 15:04:06,836 INFO [Listener at localhost/34167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:06,839 INFO [Listener at localhost/34167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:07,494 INFO [Listener at localhost/34167] util.FSUtils(471): Created version file at hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411 with version=8 2023-07-12 15:04:07,494 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/hbase-staging 2023-07-12 15:04:07,793 INFO [Listener at localhost/34167] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-12 15:04:08,246 INFO [Listener at localhost/34167] client.ConnectionUtils(127): master/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:04:08,279 INFO [Listener at localhost/34167] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:08,280 INFO [Listener at localhost/34167] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:08,280 INFO [Listener at localhost/34167] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:04:08,280 INFO [Listener at localhost/34167] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:08,280 INFO [Listener at localhost/34167] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:04:08,419 INFO [Listener at localhost/34167] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:04:08,489 DEBUG [Listener at localhost/34167] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-12 15:04:08,583 INFO [Listener at localhost/34167] ipc.NettyRpcServer(120): Bind to /172.31.10.131:46219 2023-07-12 15:04:08,593 INFO [Listener at localhost/34167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:08,595 INFO [Listener at localhost/34167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:08,615 INFO [Listener at localhost/34167] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46219 connecting to ZooKeeper ensemble=127.0.0.1:61057 2023-07-12 15:04:08,655 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:462190x0, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:04:08,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:46219-0x101c5ac4fea0000 connected 2023-07-12 15:04:08,684 DEBUG [Listener at localhost/34167] zookeeper.ZKUtil(164): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:08,685 DEBUG [Listener at localhost/34167] zookeeper.ZKUtil(164): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:08,689 DEBUG [Listener at localhost/34167] zookeeper.ZKUtil(164): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:04:08,700 DEBUG [Listener at localhost/34167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46219 2023-07-12 15:04:08,700 DEBUG [Listener at localhost/34167] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46219 2023-07-12 15:04:08,701 DEBUG [Listener at localhost/34167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46219 2023-07-12 15:04:08,701 DEBUG [Listener at localhost/34167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46219 2023-07-12 15:04:08,701 DEBUG [Listener at localhost/34167] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46219 2023-07-12 15:04:08,707 INFO [Listener at localhost/34167] master.HMaster(444): hbase.rootdir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411, hbase.cluster.distributed=false 2023-07-12 15:04:08,745 INFO [Listener at localhost/34167] client.ConnectionUtils(127): master/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:04:08,745 INFO [Listener at localhost/34167] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:08,745 INFO [Listener at localhost/34167] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:08,746 INFO [Listener at localhost/34167] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:04:08,746 INFO [Listener at localhost/34167] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:08,746 INFO [Listener at localhost/34167] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:04:08,746 INFO [Listener at localhost/34167] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:04:08,747 INFO [Listener at localhost/34167] ipc.NettyRpcServer(120): Bind to /172.31.10.131:37993 2023-07-12 15:04:08,748 INFO [Listener at localhost/34167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:08,749 INFO [Listener at localhost/34167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:08,751 INFO [Listener at localhost/34167] zookeeper.RecoverableZooKeeper(93): Process identifier=master:37993 connecting to ZooKeeper ensemble=127.0.0.1:61057 2023-07-12 15:04:08,756 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:379930x0, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:04:08,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:37993-0x101c5ac4fea0001 connected 2023-07-12 15:04:08,764 DEBUG [Listener at localhost/34167] zookeeper.ZKUtil(164): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:08,765 DEBUG [Listener at localhost/34167] zookeeper.ZKUtil(164): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:08,765 DEBUG [Listener at localhost/34167] zookeeper.ZKUtil(164): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:04:08,766 DEBUG [Listener at localhost/34167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37993 2023-07-12 15:04:08,766 DEBUG [Listener at localhost/34167] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37993 2023-07-12 15:04:08,766 DEBUG [Listener at localhost/34167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37993 2023-07-12 15:04:08,767 DEBUG [Listener at localhost/34167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37993 2023-07-12 15:04:08,767 DEBUG [Listener at localhost/34167] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37993 2023-07-12 15:04:08,767 INFO [Listener at localhost/34167] master.HMaster(444): hbase.rootdir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411, hbase.cluster.distributed=false 2023-07-12 15:04:08,799 INFO [Listener at localhost/34167] client.ConnectionUtils(127): regionserver/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:04:08,799 INFO [Listener at localhost/34167] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:08,799 INFO [Listener at localhost/34167] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:08,800 INFO [Listener at localhost/34167] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:04:08,800 INFO [Listener at localhost/34167] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:08,800 INFO [Listener at localhost/34167] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:04:08,805 INFO [Listener at localhost/34167] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:04:08,808 INFO [Listener at localhost/34167] ipc.NettyRpcServer(120): Bind to /172.31.10.131:42757 2023-07-12 15:04:08,810 INFO [Listener at localhost/34167] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 15:04:08,816 DEBUG [Listener at localhost/34167] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 15:04:08,816 INFO [Listener at localhost/34167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:08,818 INFO [Listener at localhost/34167] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:08,820 INFO [Listener at localhost/34167] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42757 connecting to ZooKeeper ensemble=127.0.0.1:61057 2023-07-12 15:04:08,824 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): regionserver:427570x0, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:04:08,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:42757-0x101c5ac4fea0002 connected 2023-07-12 15:04:08,825 DEBUG [Listener at localhost/34167] zookeeper.ZKUtil(164): regionserver:42757-0x101c5ac4fea0002, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:08,826 DEBUG [Listener at localhost/34167] zookeeper.ZKUtil(164): regionserver:42757-0x101c5ac4fea0002, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:08,826 DEBUG [Listener at localhost/34167] zookeeper.ZKUtil(164): regionserver:42757-0x101c5ac4fea0002, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:04:08,827 DEBUG [Listener at localhost/34167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42757 2023-07-12 15:04:08,827 DEBUG [Listener at localhost/34167] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42757 2023-07-12 15:04:08,827 DEBUG [Listener at localhost/34167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42757 2023-07-12 15:04:08,828 DEBUG [Listener at localhost/34167] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42757 2023-07-12 15:04:08,828 DEBUG [Listener at localhost/34167] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42757 2023-07-12 15:04:08,830 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase5.apache.org,46219,1689174247629 2023-07-12 15:04:08,830 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase5.apache.org,37993,1689174248744 2023-07-12 15:04:08,840 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:08,840 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:08,841 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase5.apache.org,37993,1689174248744 2023-07-12 15:04:08,841 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase5.apache.org,46219,1689174247629 2023-07-12 15:04:08,864 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:08,864 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): regionserver:42757-0x101c5ac4fea0002, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:08,864 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:08,864 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:08,864 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:08,865 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:08,865 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:08,866 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase5.apache.org,37993,1689174248744 from backup master directory 2023-07-12 15:04:08,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:08,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:08,866 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase5.apache.org,37993,1689174248744; waiting to become the next active master 2023-07-12 15:04:08,869 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase5.apache.org,37993,1689174248744 2023-07-12 15:04:08,869 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:08,869 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:08,870 WARN [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:04:08,870 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase5.apache.org,37993,1689174248744 2023-07-12 15:04:08,874 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-12 15:04:08,875 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-12 15:04:08,954 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/hbase.id with ID: 026bd0f4-0710-4e73-b630-af5afa9c645e 2023-07-12 15:04:08,995 INFO [master/jenkins-hbase5:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:09,008 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:09,008 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:09,045 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7fdc6cb8 to 127.0.0.1:61057 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:09,078 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1bf7e58b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:09,100 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:09,102 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 15:04:09,122 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-12 15:04:09,122 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-12 15:04:09,124 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:04:09,129 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:04:09,130 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:09,165 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store-tmp 2023-07-12 15:04:09,592 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:09,592 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:04:09,593 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:09,593 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:09,593 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:04:09,593 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:09,593 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:09,593 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:09,595 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/WALs/jenkins-hbase5.apache.org,37993,1689174248744 2023-07-12 15:04:09,615 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C37993%2C1689174248744, suffix=, logDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/WALs/jenkins-hbase5.apache.org,37993,1689174248744, archiveDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/oldWALs, maxLogs=10 2023-07-12 15:04:09,666 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45305,DS-10ff2eee-c043-4847-b4da-8d4bdc248f86,DISK] 2023-07-12 15:04:09,674 DEBUG [RS-EventLoopGroup-4-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:04:09,731 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/WALs/jenkins-hbase5.apache.org,37993,1689174248744/jenkins-hbase5.apache.org%2C37993%2C1689174248744.1689174249625 2023-07-12 15:04:09,731 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45305,DS-10ff2eee-c043-4847-b4da-8d4bdc248f86,DISK]] 2023-07-12 15:04:09,732 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:09,732 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:09,735 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:09,736 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:09,793 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:09,801 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 15:04:09,825 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 15:04:09,838 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:09,843 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:09,845 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:09,860 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:09,864 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:09,865 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70326219, jitterRate=0.047942325472831726}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:04:09,865 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:09,867 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 15:04:09,886 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:09,887 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:09,889 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:09,891 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-12 15:04:09,922 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-07-12 15:04:09,923 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 15:04:09,947 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 15:04:09,954 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 15:04:09,980 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 15:04:09,984 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 15:04:09,985 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 15:04:09,990 INFO [master/jenkins-hbase5:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 15:04:09,993 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 15:04:10,001 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:10,001 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:10,003 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 15:04:10,003 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 15:04:10,015 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 15:04:10,019 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:10,019 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): regionserver:42757-0x101c5ac4fea0002, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:10,019 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:10,019 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:10,020 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:10,020 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase5.apache.org,37993,1689174248744, sessionid=0x101c5ac4fea0001, setting cluster-up flag (Was=false) 2023-07-12 15:04:10,022 INFO [M:0;jenkins-hbase5:46219] regionserver.HRegionServer(951): ClusterId : 026bd0f4-0710-4e73-b630-af5afa9c645e 2023-07-12 15:04:10,034 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:10,034 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:10,040 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 15:04:10,041 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,37993,1689174248744 2023-07-12 15:04:10,047 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:10,047 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:10,052 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 15:04:10,053 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,37993,1689174248744 2023-07-12 15:04:10,056 WARN [master/jenkins-hbase5:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.hbase-snapshot/.tmp 2023-07-12 15:04:10,133 INFO [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(951): ClusterId : 026bd0f4-0710-4e73-b630-af5afa9c645e 2023-07-12 15:04:10,137 DEBUG [RS:0;jenkins-hbase5:42757] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 15:04:10,144 DEBUG [RS:0;jenkins-hbase5:42757] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 15:04:10,144 DEBUG [RS:0;jenkins-hbase5:42757] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 15:04:10,147 DEBUG [RS:0;jenkins-hbase5:42757] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 15:04:10,149 DEBUG [RS:0;jenkins-hbase5:42757] zookeeper.ReadOnlyZKClient(139): Connect 0x78c708ec to 127.0.0.1:61057 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:10,154 DEBUG [RS:0;jenkins-hbase5:42757] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7600e984, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:10,154 DEBUG [RS:0;jenkins-hbase5:42757] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@723f639c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:04:10,178 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:10,186 DEBUG [RS:0;jenkins-hbase5:42757] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase5:42757 2023-07-12 15:04:10,191 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:10,191 INFO [RS:0;jenkins-hbase5:42757] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 15:04:10,191 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:10,191 INFO [RS:0;jenkins-hbase5:42757] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 15:04:10,191 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:10,191 DEBUG [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 15:04:10,191 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:10,192 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase5:0, corePoolSize=10, maxPoolSize=10 2023-07-12 15:04:10,192 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:10,192 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:04:10,192 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:10,193 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174280193 2023-07-12 15:04:10,194 INFO [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase5.apache.org,37993,1689174248744 with isa=jenkins-hbase5.apache.org/172.31.10.131:42757, startcode=1689174248798 2023-07-12 15:04:10,195 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 15:04:10,197 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:10,198 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 15:04:10,203 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:10,205 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 15:04:10,212 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 15:04:10,212 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 15:04:10,212 DEBUG [RS:0;jenkins-hbase5:42757] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 15:04:10,213 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 15:04:10,213 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 15:04:10,214 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:10,216 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 15:04:10,217 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 15:04:10,217 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 15:04:10,219 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 15:04:10,219 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 15:04:10,221 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174250221,5,FailOnTimeoutGroup] 2023-07-12 15:04:10,222 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174250222,5,FailOnTimeoutGroup] 2023-07-12 15:04:10,222 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:10,222 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 15:04:10,223 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:10,224 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:10,266 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:42895, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 15:04:10,279 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=37993] master.ServerManager(394): Registering regionserver=jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:10,294 DEBUG [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411 2023-07-12 15:04:10,294 DEBUG [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:46679 2023-07-12 15:04:10,294 DEBUG [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 15:04:10,299 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:04:10,300 DEBUG [RS:0;jenkins-hbase5:42757] zookeeper.ZKUtil(162): regionserver:42757-0x101c5ac4fea0002, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:10,300 WARN [RS:0;jenkins-hbase5:42757] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:04:10,300 INFO [RS:0;jenkins-hbase5:42757] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:10,301 DEBUG [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(1948): logDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/WALs/jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:10,301 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase5.apache.org,42757,1689174248798] 2023-07-12 15:04:10,309 DEBUG [RS:0;jenkins-hbase5:42757] zookeeper.ZKUtil(162): regionserver:42757-0x101c5ac4fea0002, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:10,318 DEBUG [RS:0;jenkins-hbase5:42757] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 15:04:10,326 INFO [RS:0;jenkins-hbase5:42757] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 15:04:10,344 INFO [RS:0;jenkins-hbase5:42757] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 15:04:10,347 INFO [RS:0;jenkins-hbase5:42757] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:04:10,347 INFO [RS:0;jenkins-hbase5:42757] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:10,348 INFO [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 15:04:10,354 INFO [RS:0;jenkins-hbase5:42757] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:10,355 DEBUG [RS:0;jenkins-hbase5:42757] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:10,355 DEBUG [RS:0;jenkins-hbase5:42757] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:10,355 DEBUG [RS:0;jenkins-hbase5:42757] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:10,355 DEBUG [RS:0;jenkins-hbase5:42757] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:10,355 DEBUG [RS:0;jenkins-hbase5:42757] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:10,355 DEBUG [RS:0;jenkins-hbase5:42757] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:04:10,356 DEBUG [RS:0;jenkins-hbase5:42757] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:10,356 DEBUG [RS:0;jenkins-hbase5:42757] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:10,356 DEBUG [RS:0;jenkins-hbase5:42757] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:10,356 DEBUG [RS:0;jenkins-hbase5:42757] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:10,357 INFO [RS:0;jenkins-hbase5:42757] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:10,357 INFO [RS:0;jenkins-hbase5:42757] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:10,357 INFO [RS:0;jenkins-hbase5:42757] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:10,371 INFO [RS:0;jenkins-hbase5:42757] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 15:04:10,373 INFO [RS:0;jenkins-hbase5:42757] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,42757,1689174248798-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:10,388 INFO [RS:0;jenkins-hbase5:42757] regionserver.Replication(203): jenkins-hbase5.apache.org,42757,1689174248798 started 2023-07-12 15:04:10,388 INFO [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(1637): Serving as jenkins-hbase5.apache.org,42757,1689174248798, RpcServer on jenkins-hbase5.apache.org/172.31.10.131:42757, sessionid=0x101c5ac4fea0002 2023-07-12 15:04:10,388 DEBUG [RS:0;jenkins-hbase5:42757] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 15:04:10,389 DEBUG [RS:0;jenkins-hbase5:42757] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:10,389 DEBUG [RS:0;jenkins-hbase5:42757] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,42757,1689174248798' 2023-07-12 15:04:10,389 DEBUG [RS:0;jenkins-hbase5:42757] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:04:10,389 DEBUG [RS:0;jenkins-hbase5:42757] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:04:10,390 DEBUG [RS:0;jenkins-hbase5:42757] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 15:04:10,390 DEBUG [RS:0;jenkins-hbase5:42757] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 15:04:10,390 DEBUG [RS:0;jenkins-hbase5:42757] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:10,390 DEBUG [RS:0;jenkins-hbase5:42757] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,42757,1689174248798' 2023-07-12 15:04:10,390 DEBUG [RS:0;jenkins-hbase5:42757] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 15:04:10,390 DEBUG [RS:0;jenkins-hbase5:42757] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 15:04:10,391 DEBUG [RS:0;jenkins-hbase5:42757] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 15:04:10,391 INFO [RS:0;jenkins-hbase5:42757] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 15:04:10,391 INFO [RS:0;jenkins-hbase5:42757] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 15:04:10,501 INFO [RS:0;jenkins-hbase5:42757] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C42757%2C1689174248798, suffix=, logDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/WALs/jenkins-hbase5.apache.org,42757,1689174248798, archiveDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/oldWALs, maxLogs=32 2023-07-12 15:04:10,519 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45305,DS-10ff2eee-c043-4847-b4da-8d4bdc248f86,DISK] 2023-07-12 15:04:10,523 INFO [RS:0;jenkins-hbase5:42757] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/WALs/jenkins-hbase5.apache.org,42757,1689174248798/jenkins-hbase5.apache.org%2C42757%2C1689174248798.1689174250504 2023-07-12 15:04:10,523 DEBUG [RS:0;jenkins-hbase5:42757] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45305,DS-10ff2eee-c043-4847-b4da-8d4bdc248f86,DISK]] 2023-07-12 15:04:10,634 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:10,635 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:10,635 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411 2023-07-12 15:04:11,050 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:11,053 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 15:04:11,057 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/info 2023-07-12 15:04:11,058 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 15:04:11,059 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:11,059 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 15:04:11,061 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:04:11,062 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 15:04:11,063 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:11,063 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 15:04:11,065 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/table 2023-07-12 15:04:11,066 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 15:04:11,067 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:11,068 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740 2023-07-12 15:04:11,069 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740 2023-07-12 15:04:11,073 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 15:04:11,074 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 15:04:11,078 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:11,079 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65225638, jitterRate=-0.028062254190444946}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 15:04:11,079 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 15:04:11,079 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 15:04:11,079 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 15:04:11,079 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 15:04:11,079 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 15:04:11,079 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 15:04:11,080 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 15:04:11,080 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 15:04:11,084 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:11,084 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 15:04:11,092 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 15:04:11,105 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 15:04:11,106 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 15:04:11,258 DEBUG [jenkins-hbase5:37993] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 15:04:11,262 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,42757,1689174248798, state=OPENING 2023-07-12 15:04:11,272 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 15:04:11,274 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:11,274 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:11,275 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:04:11,275 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:04:11,277 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,42757,1689174248798}] 2023-07-12 15:04:11,459 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:11,461 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 15:04:11,465 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:57770, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 15:04:11,476 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 15:04:11,476 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:11,479 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C42757%2C1689174248798.meta, suffix=.meta, logDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/WALs/jenkins-hbase5.apache.org,42757,1689174248798, archiveDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/oldWALs, maxLogs=32 2023-07-12 15:04:11,495 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45305,DS-10ff2eee-c043-4847-b4da-8d4bdc248f86,DISK] 2023-07-12 15:04:11,499 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/WALs/jenkins-hbase5.apache.org,42757,1689174248798/jenkins-hbase5.apache.org%2C42757%2C1689174248798.meta.1689174251481.meta 2023-07-12 15:04:11,499 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45305,DS-10ff2eee-c043-4847-b4da-8d4bdc248f86,DISK]] 2023-07-12 15:04:11,500 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:11,501 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 15:04:11,517 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 15:04:11,521 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 15:04:11,527 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 15:04:11,527 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:11,527 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 15:04:11,527 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 15:04:11,530 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 15:04:11,531 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/info 2023-07-12 15:04:11,531 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/info 2023-07-12 15:04:11,532 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 15:04:11,533 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:11,533 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 15:04:11,534 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:04:11,534 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:04:11,535 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 15:04:11,536 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:11,536 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 15:04:11,537 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/table 2023-07-12 15:04:11,537 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/table 2023-07-12 15:04:11,538 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 15:04:11,538 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:11,540 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740 2023-07-12 15:04:11,542 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740 2023-07-12 15:04:11,545 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 15:04:11,547 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 15:04:11,548 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65405907, jitterRate=-0.025376036763191223}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 15:04:11,548 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 15:04:11,557 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689174251452 2023-07-12 15:04:11,574 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 15:04:11,575 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 15:04:11,576 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,42757,1689174248798, state=OPEN 2023-07-12 15:04:11,580 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 15:04:11,580 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 15:04:11,581 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:04:11,581 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:04:11,584 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 15:04:11,584 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,42757,1689174248798 in 304 msec 2023-07-12 15:04:11,589 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 15:04:11,589 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 493 msec 2023-07-12 15:04:11,593 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.4910 sec 2023-07-12 15:04:11,594 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689174251594, completionTime=-1 2023-07-12 15:04:11,594 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 15:04:11,594 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:11,655 DEBUG [hconnection-0x6446d2ec-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:04:11,659 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:57774, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:04:11,676 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:11,676 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174311676 2023-07-12 15:04:11,677 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174371677 2023-07-12 15:04:11,677 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 82 msec 2023-07-12 15:04:11,702 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,37993,1689174248744-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:11,702 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,37993,1689174248744-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:11,702 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,37993,1689174248744-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:11,704 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase5:37993, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:11,704 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:11,713 DEBUG [master/jenkins-hbase5:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 15:04:11,721 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 15:04:11,723 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:11,732 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:11,734 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:04:11,736 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:04:11,757 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/hbase/namespace/df807f39ee9b409bf5f8e7d69e53fb9c 2023-07-12 15:04:11,759 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/hbase/namespace/df807f39ee9b409bf5f8e7d69e53fb9c empty. 2023-07-12 15:04:11,760 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/hbase/namespace/df807f39ee9b409bf5f8e7d69e53fb9c 2023-07-12 15:04:11,760 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 15:04:12,182 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:12,185 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => df807f39ee9b409bf5f8e7d69e53fb9c, NAME => 'hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp 2023-07-12 15:04:12,197 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:12,197 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing df807f39ee9b409bf5f8e7d69e53fb9c, disabling compactions & flushes 2023-07-12 15:04:12,197 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c. 2023-07-12 15:04:12,197 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c. 2023-07-12 15:04:12,197 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c. after waiting 0 ms 2023-07-12 15:04:12,197 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c. 2023-07-12 15:04:12,197 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c. 2023-07-12 15:04:12,197 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for df807f39ee9b409bf5f8e7d69e53fb9c: 2023-07-12 15:04:12,202 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:04:12,223 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689174252206"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174252206"}]},"ts":"1689174252206"} 2023-07-12 15:04:12,256 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 15:04:12,258 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:04:12,263 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174252258"}]},"ts":"1689174252258"} 2023-07-12 15:04:12,267 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 15:04:12,273 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=df807f39ee9b409bf5f8e7d69e53fb9c, ASSIGN}] 2023-07-12 15:04:12,277 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=df807f39ee9b409bf5f8e7d69e53fb9c, ASSIGN 2023-07-12 15:04:12,279 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=df807f39ee9b409bf5f8e7d69e53fb9c, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,42757,1689174248798; forceNewPlan=false, retain=false 2023-07-12 15:04:12,430 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=df807f39ee9b409bf5f8e7d69e53fb9c, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:12,430 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689174252429"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174252429"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174252429"}]},"ts":"1689174252429"} 2023-07-12 15:04:12,434 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure df807f39ee9b409bf5f8e7d69e53fb9c, server=jenkins-hbase5.apache.org,42757,1689174248798}] 2023-07-12 15:04:12,593 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c. 2023-07-12 15:04:12,594 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => df807f39ee9b409bf5f8e7d69e53fb9c, NAME => 'hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:12,595 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace df807f39ee9b409bf5f8e7d69e53fb9c 2023-07-12 15:04:12,595 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:12,595 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for df807f39ee9b409bf5f8e7d69e53fb9c 2023-07-12 15:04:12,595 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for df807f39ee9b409bf5f8e7d69e53fb9c 2023-07-12 15:04:12,598 INFO [StoreOpener-df807f39ee9b409bf5f8e7d69e53fb9c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region df807f39ee9b409bf5f8e7d69e53fb9c 2023-07-12 15:04:12,600 DEBUG [StoreOpener-df807f39ee9b409bf5f8e7d69e53fb9c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/namespace/df807f39ee9b409bf5f8e7d69e53fb9c/info 2023-07-12 15:04:12,600 DEBUG [StoreOpener-df807f39ee9b409bf5f8e7d69e53fb9c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/namespace/df807f39ee9b409bf5f8e7d69e53fb9c/info 2023-07-12 15:04:12,600 INFO [StoreOpener-df807f39ee9b409bf5f8e7d69e53fb9c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region df807f39ee9b409bf5f8e7d69e53fb9c columnFamilyName info 2023-07-12 15:04:12,601 INFO [StoreOpener-df807f39ee9b409bf5f8e7d69e53fb9c-1] regionserver.HStore(310): Store=df807f39ee9b409bf5f8e7d69e53fb9c/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:12,603 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/namespace/df807f39ee9b409bf5f8e7d69e53fb9c 2023-07-12 15:04:12,604 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/namespace/df807f39ee9b409bf5f8e7d69e53fb9c 2023-07-12 15:04:12,609 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for df807f39ee9b409bf5f8e7d69e53fb9c 2023-07-12 15:04:12,613 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/namespace/df807f39ee9b409bf5f8e7d69e53fb9c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:12,614 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened df807f39ee9b409bf5f8e7d69e53fb9c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75003294, jitterRate=0.11763617396354675}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:04:12,614 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for df807f39ee9b409bf5f8e7d69e53fb9c: 2023-07-12 15:04:12,617 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c., pid=6, masterSystemTime=1689174252587 2023-07-12 15:04:12,621 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c. 2023-07-12 15:04:12,621 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c. 2023-07-12 15:04:12,622 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=df807f39ee9b409bf5f8e7d69e53fb9c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:12,623 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689174252622"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174252622"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174252622"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174252622"}]},"ts":"1689174252622"} 2023-07-12 15:04:12,631 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 15:04:12,631 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure df807f39ee9b409bf5f8e7d69e53fb9c, server=jenkins-hbase5.apache.org,42757,1689174248798 in 193 msec 2023-07-12 15:04:12,635 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 15:04:12,636 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=df807f39ee9b409bf5f8e7d69e53fb9c, ASSIGN in 358 msec 2023-07-12 15:04:12,637 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 15:04:12,637 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174252637"}]},"ts":"1689174252637"} 2023-07-12 15:04:12,641 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 15:04:12,645 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 15:04:12,649 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 921 msec 2023-07-12 15:04:12,735 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 15:04:12,738 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:04:12,738 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:12,738 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:12,777 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:12,794 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:04:12,800 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-07-12 15:04:12,810 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:12,821 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:04:12,825 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 14 msec 2023-07-12 15:04:12,835 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 15:04:12,839 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 15:04:12,839 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.968sec 2023-07-12 15:04:12,841 INFO [master/jenkins-hbase5:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 15:04:12,842 INFO [master/jenkins-hbase5:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 15:04:12,843 INFO [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 15:04:12,844 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,37993,1689174248744-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 15:04:12,844 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,37993,1689174248744-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 15:04:12,855 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 15:04:12,939 DEBUG [Listener at localhost/34167] zookeeper.ReadOnlyZKClient(139): Connect 0x66452f5c to 127.0.0.1:61057 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:12,943 DEBUG [Listener at localhost/34167] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17a154fb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:12,956 DEBUG [hconnection-0x64d6ff4d-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:04:12,967 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:57780, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:04:12,976 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase5.apache.org,37993,1689174248744 2023-07-12 15:04:12,978 WARN [Listener at localhost/34167] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-12 15:04:12,983 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:12,985 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:04:12,986 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:04:12,993 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/default/testDisableTableWithFailoverAtStep3/077e62046fb0676013bbd7f2752a109a 2023-07-12 15:04:12,993 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/default/testDisableTableWithFailoverAtStep3/737359eba122b058346d78066084ed85 2023-07-12 15:04:12,993 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/default/testDisableTableWithFailoverAtStep3/7d904a9b027be183d3dcf9d6b51ad6e9 2023-07-12 15:04:12,993 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/default/testDisableTableWithFailoverAtStep3/7455294f32f9f1b46d64ca42be444b4e 2023-07-12 15:04:12,994 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/default/testDisableTableWithFailoverAtStep3/737359eba122b058346d78066084ed85 empty. 2023-07-12 15:04:12,994 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/default/testDisableTableWithFailoverAtStep3/7455294f32f9f1b46d64ca42be444b4e empty. 2023-07-12 15:04:12,994 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/default/testDisableTableWithFailoverAtStep3/7d904a9b027be183d3dcf9d6b51ad6e9 empty. 2023-07-12 15:04:12,994 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/default/testDisableTableWithFailoverAtStep3/077e62046fb0676013bbd7f2752a109a empty. 2023-07-12 15:04:12,997 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/default/testDisableTableWithFailoverAtStep3/737359eba122b058346d78066084ed85 2023-07-12 15:04:12,997 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/default/testDisableTableWithFailoverAtStep3/7455294f32f9f1b46d64ca42be444b4e 2023-07-12 15:04:12,997 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/default/testDisableTableWithFailoverAtStep3/7d904a9b027be183d3dcf9d6b51ad6e9 2023-07-12 15:04:12,997 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/default/testDisableTableWithFailoverAtStep3/077e62046fb0676013bbd7f2752a109a 2023-07-12 15:04:12,997 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testDisableTableWithFailoverAtStep3 regions 2023-07-12 15:04:13,414 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp/data/default/testDisableTableWithFailoverAtStep3/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:13,417 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(7675): creating {ENCODED => 737359eba122b058346d78066084ed85, NAME => 'testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85.', STARTKEY => 'a', ENDKEY => 'b'}, tableDescriptor='testDisableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp 2023-07-12 15:04:13,417 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(7675): creating {ENCODED => 7d904a9b027be183d3dcf9d6b51ad6e9, NAME => 'testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9.', STARTKEY => 'b', ENDKEY => 'c'}, tableDescriptor='testDisableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp 2023-07-12 15:04:13,417 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(7675): creating {ENCODED => 077e62046fb0676013bbd7f2752a109a, NAME => 'testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a.', STARTKEY => '', ENDKEY => 'a'}, tableDescriptor='testDisableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp 2023-07-12 15:04:13,449 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:13,449 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:13,450 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1604): Closing 7d904a9b027be183d3dcf9d6b51ad6e9, disabling compactions & flushes 2023-07-12 15:04:13,450 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1604): Closing 737359eba122b058346d78066084ed85, disabling compactions & flushes 2023-07-12 15:04:13,450 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9. 2023-07-12 15:04:13,450 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85. 2023-07-12 15:04:13,450 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9. 2023-07-12 15:04:13,450 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85. 2023-07-12 15:04:13,450 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9. after waiting 0 ms 2023-07-12 15:04:13,450 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85. after waiting 0 ms 2023-07-12 15:04:13,450 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9. 2023-07-12 15:04:13,450 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85. 2023-07-12 15:04:13,451 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9. 2023-07-12 15:04:13,451 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85. 2023-07-12 15:04:13,451 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1558): Region close journal for 7d904a9b027be183d3dcf9d6b51ad6e9: 2023-07-12 15:04:13,451 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1558): Region close journal for 737359eba122b058346d78066084ed85: 2023-07-12 15:04:13,452 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(7675): creating {ENCODED => 7455294f32f9f1b46d64ca42be444b4e, NAME => 'testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e.', STARTKEY => 'c', ENDKEY => ''}, tableDescriptor='testDisableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.tmp 2023-07-12 15:04:13,850 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:13,850 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1604): Closing 077e62046fb0676013bbd7f2752a109a, disabling compactions & flushes 2023-07-12 15:04:13,850 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a. 2023-07-12 15:04:13,850 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a. 2023-07-12 15:04:13,850 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a. after waiting 0 ms 2023-07-12 15:04:13,850 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a. 2023-07-12 15:04:13,850 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a. 2023-07-12 15:04:13,850 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1558): Region close journal for 077e62046fb0676013bbd7f2752a109a: 2023-07-12 15:04:13,868 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:13,868 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1604): Closing 7455294f32f9f1b46d64ca42be444b4e, disabling compactions & flushes 2023-07-12 15:04:13,868 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e. 2023-07-12 15:04:13,869 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e. 2023-07-12 15:04:13,869 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e. after waiting 0 ms 2023-07-12 15:04:13,869 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e. 2023-07-12 15:04:13,869 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e. 2023-07-12 15:04:13,869 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1558): Region close journal for 7455294f32f9f1b46d64ca42be444b4e: 2023-07-12 15:04:13,873 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:04:13,875 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689174253874"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174253874"}]},"ts":"1689174253874"} 2023-07-12 15:04:13,875 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689174253874"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174253874"}]},"ts":"1689174253874"} 2023-07-12 15:04:13,875 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174253874"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174253874"}]},"ts":"1689174253874"} 2023-07-12 15:04:13,875 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174253874"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174253874"}]},"ts":"1689174253874"} 2023-07-12 15:04:13,920 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 4 regions to meta. 2023-07-12 15:04:13,922 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:04:13,922 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDisableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174253922"}]},"ts":"1689174253922"} 2023-07-12 15:04:13,924 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDisableTableWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-12 15:04:13,928 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=077e62046fb0676013bbd7f2752a109a, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=737359eba122b058346d78066084ed85, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7d904a9b027be183d3dcf9d6b51ad6e9, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7455294f32f9f1b46d64ca42be444b4e, ASSIGN}] 2023-07-12 15:04:13,931 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7455294f32f9f1b46d64ca42be444b4e, ASSIGN 2023-07-12 15:04:13,933 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7455294f32f9f1b46d64ca42be444b4e, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,42757,1689174248798; forceNewPlan=false, retain=false 2023-07-12 15:04:13,934 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7d904a9b027be183d3dcf9d6b51ad6e9, ASSIGN 2023-07-12 15:04:13,935 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7d904a9b027be183d3dcf9d6b51ad6e9, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,42757,1689174248798; forceNewPlan=false, retain=false 2023-07-12 15:04:13,936 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=737359eba122b058346d78066084ed85, ASSIGN 2023-07-12 15:04:13,937 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=737359eba122b058346d78066084ed85, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,42757,1689174248798; forceNewPlan=false, retain=false 2023-07-12 15:04:13,938 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=077e62046fb0676013bbd7f2752a109a, ASSIGN 2023-07-12 15:04:13,939 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=077e62046fb0676013bbd7f2752a109a, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,42757,1689174248798; forceNewPlan=false, retain=false 2023-07-12 15:04:14,084 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=077e62046fb0676013bbd7f2752a109a, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:14,084 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174254084"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174254084"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174254084"}]},"ts":"1689174254084"} 2023-07-12 15:04:14,088 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=10, state=RUNNABLE; OpenRegionProcedure 077e62046fb0676013bbd7f2752a109a, server=jenkins-hbase5.apache.org,42757,1689174248798}] 2023-07-12 15:04:14,092 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=737359eba122b058346d78066084ed85, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:14,092 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689174254092"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174254092"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174254092"}]},"ts":"1689174254092"} 2023-07-12 15:04:14,095 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=11, state=RUNNABLE; OpenRegionProcedure 737359eba122b058346d78066084ed85, server=jenkins-hbase5.apache.org,42757,1689174248798}] 2023-07-12 15:04:14,099 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=7d904a9b027be183d3dcf9d6b51ad6e9, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:14,099 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689174254099"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174254099"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174254099"}]},"ts":"1689174254099"} 2023-07-12 15:04:14,102 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=12, state=RUNNABLE; OpenRegionProcedure 7d904a9b027be183d3dcf9d6b51ad6e9, server=jenkins-hbase5.apache.org,42757,1689174248798}] 2023-07-12 15:04:14,106 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=7455294f32f9f1b46d64ca42be444b4e, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:14,106 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174254106"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174254106"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174254106"}]},"ts":"1689174254106"} 2023-07-12 15:04:14,108 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=13, state=RUNNABLE; OpenRegionProcedure 7455294f32f9f1b46d64ca42be444b4e, server=jenkins-hbase5.apache.org,42757,1689174248798}] 2023-07-12 15:04:14,247 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a. 2023-07-12 15:04:14,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 077e62046fb0676013bbd7f2752a109a, NAME => 'testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a.', STARTKEY => '', ENDKEY => 'a'} 2023-07-12 15:04:14,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDisableTableWithFailoverAtStep3 077e62046fb0676013bbd7f2752a109a 2023-07-12 15:04:14,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:14,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 077e62046fb0676013bbd7f2752a109a 2023-07-12 15:04:14,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 077e62046fb0676013bbd7f2752a109a 2023-07-12 15:04:14,250 INFO [StoreOpener-077e62046fb0676013bbd7f2752a109a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 077e62046fb0676013bbd7f2752a109a 2023-07-12 15:04:14,252 DEBUG [StoreOpener-077e62046fb0676013bbd7f2752a109a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/077e62046fb0676013bbd7f2752a109a/f1 2023-07-12 15:04:14,252 DEBUG [StoreOpener-077e62046fb0676013bbd7f2752a109a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/077e62046fb0676013bbd7f2752a109a/f1 2023-07-12 15:04:14,253 INFO [StoreOpener-077e62046fb0676013bbd7f2752a109a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 077e62046fb0676013bbd7f2752a109a columnFamilyName f1 2023-07-12 15:04:14,254 INFO [StoreOpener-077e62046fb0676013bbd7f2752a109a-1] regionserver.HStore(310): Store=077e62046fb0676013bbd7f2752a109a/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:14,254 INFO [StoreOpener-077e62046fb0676013bbd7f2752a109a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 077e62046fb0676013bbd7f2752a109a 2023-07-12 15:04:14,256 DEBUG [StoreOpener-077e62046fb0676013bbd7f2752a109a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/077e62046fb0676013bbd7f2752a109a/f2 2023-07-12 15:04:14,256 DEBUG [StoreOpener-077e62046fb0676013bbd7f2752a109a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/077e62046fb0676013bbd7f2752a109a/f2 2023-07-12 15:04:14,257 INFO [StoreOpener-077e62046fb0676013bbd7f2752a109a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 077e62046fb0676013bbd7f2752a109a columnFamilyName f2 2023-07-12 15:04:14,257 INFO [StoreOpener-077e62046fb0676013bbd7f2752a109a-1] regionserver.HStore(310): Store=077e62046fb0676013bbd7f2752a109a/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:14,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/077e62046fb0676013bbd7f2752a109a 2023-07-12 15:04:14,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/077e62046fb0676013bbd7f2752a109a 2023-07-12 15:04:14,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testDisableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:04:14,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 077e62046fb0676013bbd7f2752a109a 2023-07-12 15:04:14,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/077e62046fb0676013bbd7f2752a109a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:14,269 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 077e62046fb0676013bbd7f2752a109a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72700653, jitterRate=0.08332414925098419}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:04:14,269 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 077e62046fb0676013bbd7f2752a109a: 2023-07-12 15:04:14,270 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a., pid=14, masterSystemTime=1689174254241 2023-07-12 15:04:14,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a. 2023-07-12 15:04:14,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a. 2023-07-12 15:04:14,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85. 2023-07-12 15:04:14,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 737359eba122b058346d78066084ed85, NAME => 'testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85.', STARTKEY => 'a', ENDKEY => 'b'} 2023-07-12 15:04:14,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDisableTableWithFailoverAtStep3 737359eba122b058346d78066084ed85 2023-07-12 15:04:14,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:14,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 737359eba122b058346d78066084ed85 2023-07-12 15:04:14,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 737359eba122b058346d78066084ed85 2023-07-12 15:04:14,273 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=077e62046fb0676013bbd7f2752a109a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:14,274 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174254273"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174254273"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174254273"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174254273"}]},"ts":"1689174254273"} 2023-07-12 15:04:14,275 INFO [StoreOpener-737359eba122b058346d78066084ed85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 737359eba122b058346d78066084ed85 2023-07-12 15:04:14,277 DEBUG [StoreOpener-737359eba122b058346d78066084ed85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/737359eba122b058346d78066084ed85/f1 2023-07-12 15:04:14,277 DEBUG [StoreOpener-737359eba122b058346d78066084ed85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/737359eba122b058346d78066084ed85/f1 2023-07-12 15:04:14,277 INFO [StoreOpener-737359eba122b058346d78066084ed85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 737359eba122b058346d78066084ed85 columnFamilyName f1 2023-07-12 15:04:14,278 INFO [StoreOpener-737359eba122b058346d78066084ed85-1] regionserver.HStore(310): Store=737359eba122b058346d78066084ed85/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:14,278 INFO [StoreOpener-737359eba122b058346d78066084ed85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 737359eba122b058346d78066084ed85 2023-07-12 15:04:14,281 DEBUG [StoreOpener-737359eba122b058346d78066084ed85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/737359eba122b058346d78066084ed85/f2 2023-07-12 15:04:14,281 DEBUG [StoreOpener-737359eba122b058346d78066084ed85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/737359eba122b058346d78066084ed85/f2 2023-07-12 15:04:14,281 INFO [StoreOpener-737359eba122b058346d78066084ed85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 737359eba122b058346d78066084ed85 columnFamilyName f2 2023-07-12 15:04:14,282 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=10 2023-07-12 15:04:14,283 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=10, state=SUCCESS; OpenRegionProcedure 077e62046fb0676013bbd7f2752a109a, server=jenkins-hbase5.apache.org,42757,1689174248798 in 188 msec 2023-07-12 15:04:14,283 INFO [StoreOpener-737359eba122b058346d78066084ed85-1] regionserver.HStore(310): Store=737359eba122b058346d78066084ed85/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:14,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/737359eba122b058346d78066084ed85 2023-07-12 15:04:14,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/737359eba122b058346d78066084ed85 2023-07-12 15:04:14,286 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=077e62046fb0676013bbd7f2752a109a, ASSIGN in 355 msec 2023-07-12 15:04:14,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testDisableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:04:14,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 737359eba122b058346d78066084ed85 2023-07-12 15:04:14,294 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/737359eba122b058346d78066084ed85/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:14,295 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 737359eba122b058346d78066084ed85; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72688999, jitterRate=0.08315049111843109}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:04:14,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 737359eba122b058346d78066084ed85: 2023-07-12 15:04:14,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85., pid=15, masterSystemTime=1689174254241 2023-07-12 15:04:14,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85. 2023-07-12 15:04:14,298 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85. 2023-07-12 15:04:14,298 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e. 2023-07-12 15:04:14,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7455294f32f9f1b46d64ca42be444b4e, NAME => 'testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e.', STARTKEY => 'c', ENDKEY => ''} 2023-07-12 15:04:14,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDisableTableWithFailoverAtStep3 7455294f32f9f1b46d64ca42be444b4e 2023-07-12 15:04:14,299 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=737359eba122b058346d78066084ed85, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:14,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:14,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 7455294f32f9f1b46d64ca42be444b4e 2023-07-12 15:04:14,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 7455294f32f9f1b46d64ca42be444b4e 2023-07-12 15:04:14,299 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689174254299"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174254299"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174254299"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174254299"}]},"ts":"1689174254299"} 2023-07-12 15:04:14,301 INFO [StoreOpener-7455294f32f9f1b46d64ca42be444b4e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 7455294f32f9f1b46d64ca42be444b4e 2023-07-12 15:04:14,303 DEBUG [StoreOpener-7455294f32f9f1b46d64ca42be444b4e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/7455294f32f9f1b46d64ca42be444b4e/f1 2023-07-12 15:04:14,303 DEBUG [StoreOpener-7455294f32f9f1b46d64ca42be444b4e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/7455294f32f9f1b46d64ca42be444b4e/f1 2023-07-12 15:04:14,303 INFO [StoreOpener-7455294f32f9f1b46d64ca42be444b4e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7455294f32f9f1b46d64ca42be444b4e columnFamilyName f1 2023-07-12 15:04:14,304 INFO [StoreOpener-7455294f32f9f1b46d64ca42be444b4e-1] regionserver.HStore(310): Store=7455294f32f9f1b46d64ca42be444b4e/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:14,305 INFO [StoreOpener-7455294f32f9f1b46d64ca42be444b4e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 7455294f32f9f1b46d64ca42be444b4e 2023-07-12 15:04:14,305 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=11 2023-07-12 15:04:14,306 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=11, state=SUCCESS; OpenRegionProcedure 737359eba122b058346d78066084ed85, server=jenkins-hbase5.apache.org,42757,1689174248798 in 207 msec 2023-07-12 15:04:14,307 DEBUG [StoreOpener-7455294f32f9f1b46d64ca42be444b4e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/7455294f32f9f1b46d64ca42be444b4e/f2 2023-07-12 15:04:14,307 DEBUG [StoreOpener-7455294f32f9f1b46d64ca42be444b4e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/7455294f32f9f1b46d64ca42be444b4e/f2 2023-07-12 15:04:14,308 INFO [StoreOpener-7455294f32f9f1b46d64ca42be444b4e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7455294f32f9f1b46d64ca42be444b4e columnFamilyName f2 2023-07-12 15:04:14,308 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=737359eba122b058346d78066084ed85, ASSIGN in 378 msec 2023-07-12 15:04:14,308 INFO [StoreOpener-7455294f32f9f1b46d64ca42be444b4e-1] regionserver.HStore(310): Store=7455294f32f9f1b46d64ca42be444b4e/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:14,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/7455294f32f9f1b46d64ca42be444b4e 2023-07-12 15:04:14,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/7455294f32f9f1b46d64ca42be444b4e 2023-07-12 15:04:14,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testDisableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:04:14,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 7455294f32f9f1b46d64ca42be444b4e 2023-07-12 15:04:14,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/7455294f32f9f1b46d64ca42be444b4e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:14,319 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 7455294f32f9f1b46d64ca42be444b4e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66490499, jitterRate=-0.009214356541633606}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:04:14,320 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 7455294f32f9f1b46d64ca42be444b4e: 2023-07-12 15:04:14,321 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e., pid=17, masterSystemTime=1689174254241 2023-07-12 15:04:14,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e. 2023-07-12 15:04:14,323 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e. 2023-07-12 15:04:14,323 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9. 2023-07-12 15:04:14,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7d904a9b027be183d3dcf9d6b51ad6e9, NAME => 'testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9.', STARTKEY => 'b', ENDKEY => 'c'} 2023-07-12 15:04:14,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDisableTableWithFailoverAtStep3 7d904a9b027be183d3dcf9d6b51ad6e9 2023-07-12 15:04:14,324 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=7455294f32f9f1b46d64ca42be444b4e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:14,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:14,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 7d904a9b027be183d3dcf9d6b51ad6e9 2023-07-12 15:04:14,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 7d904a9b027be183d3dcf9d6b51ad6e9 2023-07-12 15:04:14,324 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174254323"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174254323"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174254323"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174254323"}]},"ts":"1689174254323"} 2023-07-12 15:04:14,325 INFO [StoreOpener-7d904a9b027be183d3dcf9d6b51ad6e9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 7d904a9b027be183d3dcf9d6b51ad6e9 2023-07-12 15:04:14,328 DEBUG [StoreOpener-7d904a9b027be183d3dcf9d6b51ad6e9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/7d904a9b027be183d3dcf9d6b51ad6e9/f1 2023-07-12 15:04:14,328 DEBUG [StoreOpener-7d904a9b027be183d3dcf9d6b51ad6e9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/7d904a9b027be183d3dcf9d6b51ad6e9/f1 2023-07-12 15:04:14,328 INFO [StoreOpener-7d904a9b027be183d3dcf9d6b51ad6e9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7d904a9b027be183d3dcf9d6b51ad6e9 columnFamilyName f1 2023-07-12 15:04:14,329 INFO [StoreOpener-7d904a9b027be183d3dcf9d6b51ad6e9-1] regionserver.HStore(310): Store=7d904a9b027be183d3dcf9d6b51ad6e9/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:14,329 INFO [StoreOpener-7d904a9b027be183d3dcf9d6b51ad6e9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 7d904a9b027be183d3dcf9d6b51ad6e9 2023-07-12 15:04:14,330 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=13 2023-07-12 15:04:14,330 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=13, state=SUCCESS; OpenRegionProcedure 7455294f32f9f1b46d64ca42be444b4e, server=jenkins-hbase5.apache.org,42757,1689174248798 in 219 msec 2023-07-12 15:04:14,332 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7455294f32f9f1b46d64ca42be444b4e, ASSIGN in 402 msec 2023-07-12 15:04:14,333 DEBUG [StoreOpener-7d904a9b027be183d3dcf9d6b51ad6e9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/7d904a9b027be183d3dcf9d6b51ad6e9/f2 2023-07-12 15:04:14,333 DEBUG [StoreOpener-7d904a9b027be183d3dcf9d6b51ad6e9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/7d904a9b027be183d3dcf9d6b51ad6e9/f2 2023-07-12 15:04:14,334 INFO [StoreOpener-7d904a9b027be183d3dcf9d6b51ad6e9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7d904a9b027be183d3dcf9d6b51ad6e9 columnFamilyName f2 2023-07-12 15:04:14,336 INFO [StoreOpener-7d904a9b027be183d3dcf9d6b51ad6e9-1] regionserver.HStore(310): Store=7d904a9b027be183d3dcf9d6b51ad6e9/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:14,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/7d904a9b027be183d3dcf9d6b51ad6e9 2023-07-12 15:04:14,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/7d904a9b027be183d3dcf9d6b51ad6e9 2023-07-12 15:04:14,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testDisableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:04:14,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 7d904a9b027be183d3dcf9d6b51ad6e9 2023-07-12 15:04:14,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/7d904a9b027be183d3dcf9d6b51ad6e9/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:14,346 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 7d904a9b027be183d3dcf9d6b51ad6e9; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59755751, jitterRate=-0.10956992208957672}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:04:14,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 7d904a9b027be183d3dcf9d6b51ad6e9: 2023-07-12 15:04:14,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9., pid=16, masterSystemTime=1689174254241 2023-07-12 15:04:14,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9. 2023-07-12 15:04:14,349 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9. 2023-07-12 15:04:14,350 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=7d904a9b027be183d3dcf9d6b51ad6e9, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:14,350 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689174254349"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174254349"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174254349"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174254349"}]},"ts":"1689174254349"} 2023-07-12 15:04:14,355 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=12 2023-07-12 15:04:14,355 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=12, state=SUCCESS; OpenRegionProcedure 7d904a9b027be183d3dcf9d6b51ad6e9, server=jenkins-hbase5.apache.org,42757,1689174248798 in 251 msec 2023-07-12 15:04:14,358 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=12, resume processing ppid=9 2023-07-12 15:04:14,358 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7d904a9b027be183d3dcf9d6b51ad6e9, ASSIGN in 427 msec 2023-07-12 15:04:14,359 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 15:04:14,360 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDisableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174254359"}]},"ts":"1689174254359"} 2023-07-12 15:04:14,361 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDisableTableWithFailoverAtStep3, state=ENABLED in hbase:meta 2023-07-12 15:04:14,368 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 15:04:14,371 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 in 1.3870 sec 2023-07-12 15:04:14,483 WARN [Listener at localhost/34167] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-07-12 15:04:14,490 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:14,492 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:14,492 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:14,492 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:14,493 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:14,740 INFO [Listener at localhost/34167] procedure.TestMasterFailoverWithProcedures(306): Restart 0 exec state: pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:14,742 INFO [Listener at localhost/34167] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:04:14,742 INFO [Listener at localhost/34167] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:14,742 INFO [Listener at localhost/34167] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:14,743 ERROR [Listener at localhost/34167] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] 2023-07-12 15:04:14,744 INFO [Listener at localhost/34167] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:04:14,744 INFO [Listener at localhost/34167] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:14,744 INFO [Listener at localhost/34167] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:14,744 INFO [Listener at localhost/34167] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:14,745 INFO [Listener at localhost/34167] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-12 15:04:14,748 DEBUG [Listener at localhost/34167] store.ProcedureTree(114): Procedure Procedure(pid=18, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DisableTableProcedure) stack ids=[] 2023-07-12 15:04:14,749 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:14,750 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:14,750 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:14,750 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:14,751 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:14,751 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(468): Loading pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:14,751 DEBUG [Listener at localhost/34167] procedure2.Procedure(966): pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:14,752 INFO [Listener at localhost/34167] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-07-12 15:04:14,753 DEBUG [Listener at localhost/34167] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:14,753 DEBUG [Listener at localhost/34167] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,42757,1689174248798, table=hbase:meta, region=1588230740 2023-07-12 15:04:14,754 INFO [Listener at localhost/34167] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174284754 2023-07-12 15:04:14,755 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:14,755 DEBUG [Listener at localhost/34167] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:14,759 DEBUG [Listener at localhost/34167] assignment.RegionStateStore(147): Load hbase:meta entry region=df807f39ee9b409bf5f8e7d69e53fb9c, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:14,760 DEBUG [Listener at localhost/34167] assignment.RegionStateStore(147): Load hbase:meta entry region=077e62046fb0676013bbd7f2752a109a, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:14,761 DEBUG [Listener at localhost/34167] assignment.RegionStateStore(147): Load hbase:meta entry region=737359eba122b058346d78066084ed85, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:14,761 DEBUG [Listener at localhost/34167] assignment.RegionStateStore(147): Load hbase:meta entry region=7d904a9b027be183d3dcf9d6b51ad6e9, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:14,762 DEBUG [Listener at localhost/34167] assignment.RegionStateStore(147): Load hbase:meta entry region=7455294f32f9f1b46d64ca42be444b4e, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:14,762 INFO [Listener at localhost/34167] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:14,762 INFO [Listener at localhost/34167] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174314762 2023-07-12 15:04:14,762 INFO [Listener at localhost/34167] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174374762 2023-07-12 15:04:14,762 INFO [Listener at localhost/34167] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-07-12 15:04:14,762 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:04:14,764 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:14,764 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_SET_DISABLING_TABLE_STATE, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:14,764 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:14,764 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_SET_DISABLING_TABLE_STATE, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:15,012 INFO [Listener at localhost/34167] procedure.TestMasterFailoverWithProcedures(306): Restart 1 exec state: pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:15,012 INFO [Listener at localhost/34167] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:04:15,012 INFO [Listener at localhost/34167] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:15,013 INFO [Listener at localhost/34167] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:15,013 ERROR [Listener at localhost/34167] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:15,013 INFO [Listener at localhost/34167] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:04:15,013 INFO [Listener at localhost/34167] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:15,013 INFO [Listener at localhost/34167] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:15,013 INFO [Listener at localhost/34167] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:15,014 INFO [Listener at localhost/34167] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-12 15:04:15,016 DEBUG [Listener at localhost/34167] store.ProcedureTree(114): Procedure Procedure(pid=18, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DisableTableProcedure) stack ids=[0] 2023-07-12 15:04:15,017 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:15,017 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:15,018 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:15,018 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:15,018 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:15,019 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(468): Loading pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:15,019 DEBUG [Listener at localhost/34167] procedure2.Procedure(966): pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:15,019 INFO [Listener at localhost/34167] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-07-12 15:04:15,020 DEBUG [Listener at localhost/34167] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:15,020 DEBUG [Listener at localhost/34167] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,42757,1689174248798, table=hbase:meta, region=1588230740 2023-07-12 15:04:15,021 INFO [Listener at localhost/34167] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174285020 2023-07-12 15:04:15,046 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:15,046 DEBUG [Listener at localhost/34167] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:15,053 DEBUG [Listener at localhost/34167] assignment.RegionStateStore(147): Load hbase:meta entry region=df807f39ee9b409bf5f8e7d69e53fb9c, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:15,054 DEBUG [Listener at localhost/34167] assignment.RegionStateStore(147): Load hbase:meta entry region=077e62046fb0676013bbd7f2752a109a, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:15,054 DEBUG [Listener at localhost/34167] assignment.RegionStateStore(147): Load hbase:meta entry region=737359eba122b058346d78066084ed85, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:15,055 DEBUG [Listener at localhost/34167] assignment.RegionStateStore(147): Load hbase:meta entry region=7d904a9b027be183d3dcf9d6b51ad6e9, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:15,055 DEBUG [Listener at localhost/34167] assignment.RegionStateStore(147): Load hbase:meta entry region=7455294f32f9f1b46d64ca42be444b4e, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:15,055 INFO [Listener at localhost/34167] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:15,055 INFO [Listener at localhost/34167] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174315055 2023-07-12 15:04:15,056 INFO [Listener at localhost/34167] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174375056 2023-07-12 15:04:15,056 INFO [Listener at localhost/34167] assignment.AssignmentManager(1536): Joined the cluster in 9 msec 2023-07-12 15:04:15,056 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:04:15,058 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDisableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174255058"}]},"ts":"1689174255058"} 2023-07-12 15:04:15,060 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDisableTableWithFailoverAtStep3, state=DISABLING in hbase:meta 2023-07-12 15:04:15,062 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testDisableTableWithFailoverAtStep3 to state=DISABLING 2023-07-12 15:04:15,062 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:15,062 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_MARK_REGIONS_OFFLINE, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:15,062 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:15,062 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_MARK_REGIONS_OFFLINE, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:15,306 INFO [Listener at localhost/34167] procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state: pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:15,306 INFO [Listener at localhost/34167] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:04:15,306 INFO [Listener at localhost/34167] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:15,306 INFO [Listener at localhost/34167] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:15,307 ERROR [Listener at localhost/34167] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:15,307 INFO [Listener at localhost/34167] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:04:15,307 INFO [Listener at localhost/34167] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:15,308 INFO [Listener at localhost/34167] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:15,308 INFO [Listener at localhost/34167] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:15,309 INFO [Listener at localhost/34167] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-12 15:04:15,311 DEBUG [Listener at localhost/34167] store.ProcedureTree(114): Procedure Procedure(pid=18, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DisableTableProcedure) stack ids=[0, 1] 2023-07-12 15:04:15,313 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:15,314 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:15,314 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:15,314 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:15,315 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:15,315 DEBUG [Listener at localhost/34167] procedure2.ProcedureExecutor(468): Loading pid=18, state=RUNNABLE:DISABLE_TABLE_SET_DISABLING_TABLE_STATE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:15,315 DEBUG [Listener at localhost/34167] procedure2.Procedure(966): pid=18, state=RUNNABLE:DISABLE_TABLE_SET_DISABLING_TABLE_STATE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:15,315 INFO [Listener at localhost/34167] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-07-12 15:04:15,316 DEBUG [Listener at localhost/34167] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:15,317 DEBUG [Listener at localhost/34167] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,42757,1689174248798, table=hbase:meta, region=1588230740 2023-07-12 15:04:15,317 INFO [Listener at localhost/34167] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174285317 2023-07-12 15:04:15,318 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:DISABLE_TABLE_SET_DISABLING_TABLE_STATE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:15,319 DEBUG [Listener at localhost/34167] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:15,322 DEBUG [Listener at localhost/34167] assignment.RegionStateStore(147): Load hbase:meta entry region=df807f39ee9b409bf5f8e7d69e53fb9c, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:15,323 DEBUG [Listener at localhost/34167] assignment.RegionStateStore(147): Load hbase:meta entry region=077e62046fb0676013bbd7f2752a109a, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:15,323 DEBUG [Listener at localhost/34167] assignment.RegionStateStore(147): Load hbase:meta entry region=737359eba122b058346d78066084ed85, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:15,324 DEBUG [Listener at localhost/34167] assignment.RegionStateStore(147): Load hbase:meta entry region=7d904a9b027be183d3dcf9d6b51ad6e9, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:15,324 DEBUG [Listener at localhost/34167] assignment.RegionStateStore(147): Load hbase:meta entry region=7455294f32f9f1b46d64ca42be444b4e, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:15,324 INFO [Listener at localhost/34167] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:15,324 INFO [Listener at localhost/34167] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174315324 2023-07-12 15:04:15,324 INFO [Listener at localhost/34167] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174375324 2023-07-12 15:04:15,324 INFO [Listener at localhost/34167] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-07-12 15:04:15,325 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDisableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174255325"}]},"ts":"1689174255325"} 2023-07-12 15:04:15,327 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDisableTableWithFailoverAtStep3, state=DISABLING in hbase:meta 2023-07-12 15:04:15,330 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testDisableTableWithFailoverAtStep3 to state=DISABLING 2023-07-12 15:04:15,330 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:04:15,333 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=077e62046fb0676013bbd7f2752a109a, UNASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=737359eba122b058346d78066084ed85, UNASSIGN}, {pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7d904a9b027be183d3dcf9d6b51ad6e9, UNASSIGN}, {pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7455294f32f9f1b46d64ca42be444b4e, UNASSIGN}] 2023-07-12 15:04:15,333 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:15,333 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=18, state=WAITING:DISABLE_TABLE_ADD_REPLICATION_BARRIER, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:15,333 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:15,333 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=WAITING:DISABLE_TABLE_ADD_REPLICATION_BARRIER, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:15,575 INFO [Listener at localhost/34167] procedure.TestMasterFailoverWithProcedures(313): Trigger master failover 2023-07-12 15:04:15,575 INFO [Listener at localhost/34167] hbase.MiniHBaseCluster(607): Aborting jenkins-hbase5.apache.org,37993,1689174248744 2023-07-12 15:04:15,578 ERROR [Listener at localhost/34167] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 15:04:15,579 ERROR [Listener at localhost/34167] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase5.apache.org,37993,1689174248744: Aborting for tests ***** java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:608) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:411) at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.masterFailover(MasterProcedureTestingUtility.java:135) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testRecoveryAndDoubleExecution(TestMasterFailoverWithProcedures.java:314) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testDisableTableWithFailoverAtStep(TestMasterFailoverWithProcedures.java:254) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testDisableTableWithFailover(TestMasterFailoverWithProcedures.java:235) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:04:15,579 INFO [Listener at localhost/34167] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,37993,1689174248744' ***** 2023-07-12 15:04:15,579 INFO [Listener at localhost/34167] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost/34167 2023-07-12 15:04:15,579 DEBUG [M:1;jenkins-hbase5:37993] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42343ea2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:04:15,580 INFO [M:1;jenkins-hbase5:37993] regionserver.HRegionServer(1141): aborting server jenkins-hbase5.apache.org,37993,1689174248744 2023-07-12 15:04:15,581 DEBUG [M:1;jenkins-hbase5:37993] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7fdc6cb8 to 127.0.0.1:61057 2023-07-12 15:04:15,581 DEBUG [M:1;jenkins-hbase5:37993] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:15,582 INFO [M:1;jenkins-hbase5:37993] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,37993,1689174248744; all regions closed. 2023-07-12 15:04:15,582 DEBUG [M:1;jenkins-hbase5:37993] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:15,582 DEBUG [M:1;jenkins-hbase5:37993] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 15:04:15,582 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 15:04:15,583 DEBUG [M:1;jenkins-hbase5:37993] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 15:04:15,583 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174250222] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174250222,5,FailOnTimeoutGroup] 2023-07-12 15:04:15,584 INFO [M:1;jenkins-hbase5:37993] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 15:04:15,583 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174250221] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174250221,5,FailOnTimeoutGroup] 2023-07-12 15:04:15,584 INFO [M:1;jenkins-hbase5:37993] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 15:04:15,584 INFO [M:1;jenkins-hbase5:37993] hbase.ChoreService(369): Chore service for: master/jenkins-hbase5:0 had [] on shutdown 2023-07-12 15:04:15,585 DEBUG [M:1;jenkins-hbase5:37993] master.HMaster(1512): Stopping service threads 2023-07-12 15:04:15,585 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:15,585 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:15,585 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): regionserver:42757-0x101c5ac4fea0002, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:15,585 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:15,585 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:15,585 INFO [M:1;jenkins-hbase5:37993] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 15:04:15,586 ERROR [M:1;jenkins-hbase5:37993] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:15,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42757-0x101c5ac4fea0002, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:15,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:15,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:15,586 INFO [M:1;jenkins-hbase5:37993] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 15:04:15,587 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 15:04:15,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:15,587 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-12 15:04:15,587 DEBUG [M:1;jenkins-hbase5:37993] zookeeper.ZKUtil(398): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 15:04:15,587 WARN [M:1;jenkins-hbase5:37993] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 15:04:15,587 INFO [M:1;jenkins-hbase5:37993] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:15,588 INFO [M:1;jenkins-hbase5:37993] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 15:04:15,588 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:15,588 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:15,588 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:15,588 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): regionserver:42757-0x101c5ac4fea0002, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:15,588 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:15,590 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:15,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:15,590 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase5.apache.org,46219,1689174247629 from backup master directory 2023-07-12 15:04:15,592 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase5.apache.org,46219,1689174247629 2023-07-12 15:04:15,592 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:15,592 WARN [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:04:15,592 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:15,592 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase5.apache.org,46219,1689174247629 2023-07-12 15:04:15,601 DEBUG [M:1;jenkins-hbase5:37993] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:04:15,601 INFO [M:1;jenkins-hbase5:37993] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:15,601 DEBUG [M:1;jenkins-hbase5:37993] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:15,601 DEBUG [M:1;jenkins-hbase5:37993] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:04:15,601 DEBUG [M:1;jenkins-hbase5:37993] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:15,601 ERROR [M:1;jenkins-hbase5:37993] regionserver.HRegion(1825): Memstore data size is 63955 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:15,602 INFO [M:1;jenkins-hbase5:37993] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:15,602 DEBUG [M:1;jenkins-hbase5:37993] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:15,602 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:04:15,602 INFO [M:1;jenkins-hbase5:37993] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 15:04:15,603 INFO [M:1;jenkins-hbase5:37993] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:37993 2023-07-12 15:04:15,611 DEBUG [M:1;jenkins-hbase5:37993] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase5.apache.org,37993,1689174248744 already deleted, retry=false 2023-07-12 15:04:15,617 INFO [master/jenkins-hbase5:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:15,643 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x41579235 to 127.0.0.1:61057 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:15,646 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50646037, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:15,647 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:15,647 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 15:04:15,648 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:15,656 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/WALs/jenkins-hbase5.apache.org,37993,1689174248744 to hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/WALs/jenkins-hbase5.apache.org,37993,1689174248744-dead as it is dead 2023-07-12 15:04:15,658 INFO [master/jenkins-hbase5:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/WALs/jenkins-hbase5.apache.org,37993,1689174248744-dead/jenkins-hbase5.apache.org%2C37993%2C1689174248744.1689174249625 2023-07-12 15:04:15,663 INFO [master/jenkins-hbase5:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/WALs/jenkins-hbase5.apache.org,37993,1689174248744-dead/jenkins-hbase5.apache.org%2C37993%2C1689174248744.1689174249625 after 5ms 2023-07-12 15:04:15,664 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/WALs/jenkins-hbase5.apache.org,37993,1689174248744-dead/jenkins-hbase5.apache.org%2C37993%2C1689174248744.1689174249625 to hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C37993%2C1689174248744.1689174249625 2023-07-12 15:04:15,664 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/WALs/jenkins-hbase5.apache.org,37993,1689174248744-dead 2023-07-12 15:04:15,665 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/WALs/jenkins-hbase5.apache.org,46219,1689174247629 2023-07-12 15:04:15,667 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C46219%2C1689174247629, suffix=, logDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/WALs/jenkins-hbase5.apache.org,46219,1689174247629, archiveDir=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/oldWALs, maxLogs=10 2023-07-12 15:04:15,683 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45305,DS-10ff2eee-c043-4847-b4da-8d4bdc248f86,DISK] 2023-07-12 15:04:15,686 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/WALs/jenkins-hbase5.apache.org,46219,1689174247629/jenkins-hbase5.apache.org%2C46219%2C1689174247629.1689174255668 2023-07-12 15:04:15,686 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45305,DS-10ff2eee-c043-4847-b4da-8d4bdc248f86,DISK]] 2023-07-12 15:04:15,686 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:15,687 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:15,687 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:15,687 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:15,688 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:15,689 WARN [RS-EventLoopGroup-4-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase5.apache.org/172.31.10.131:37993 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:37993 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:04:15,690 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 15:04:15,690 DEBUG [RS-EventLoopGroup-4-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase5.apache.org/172.31.10.131:37993 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:37993 2023-07-12 15:04:15,690 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 15:04:15,691 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:15,692 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-12 15:04:15,692 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C37993%2C1689174248744.1689174249625 2023-07-12 15:04:15,713 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 156, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=134, path=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C37993%2C1689174248744.1689174249625 2023-07-12 15:04:15,714 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=62.46 KB heapSize=74.24 KB 2023-07-12 15:04:15,783 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:15,784 INFO [M:1;jenkins-hbase5:37993] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,37993,1689174248744; zookeeper connection closed. 2023-07-12 15:04:15,784 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:37993-0x101c5ac4fea0001, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:15,792 DEBUG [RS:0;jenkins-hbase5:42757] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 15:04:15,795 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:57015, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 15:04:15,795 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46219] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:15,914 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46219] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:16,017 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46219] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:16,120 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46219] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:16,167 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=62.46 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/407d9c8db55d44e8b5f82ea57905ef70 2023-07-12 15:04:16,206 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/407d9c8db55d44e8b5f82ea57905ef70 as hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/407d9c8db55d44e8b5f82ea57905ef70 2023-07-12 15:04:16,215 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/407d9c8db55d44e8b5f82ea57905ef70, entries=156, sequenceid=134, filesize=67.4 K 2023-07-12 15:04:16,215 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~62.46 KB/63955, heapSize ~74.23 KB/76008, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 502ms, sequenceid=134, compaction requested=false; wal=null 2023-07-12 15:04:16,217 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C37993%2C1689174248744.1689174249625 2023-07-12 15:04:16,221 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:16,222 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46219] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:16,225 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/134.seqid, newMaxSeqId=134, maxSeqId=1 2023-07-12 15:04:16,226 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=135; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62839033, jitterRate=-0.06362544000148773}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:04:16,226 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:16,227 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 15:04:16,228 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:16,228 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:16,228 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:16,228 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:16,240 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] store.ProcedureTree(114): Procedure Procedure(pid=18, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DisableTableProcedure) stack ids=[0, 1, 2] 2023-07-12 15:04:16,241 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:16,242 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:16,242 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:16,242 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:16,243 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:16,243 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(468): Loading pid=18, state=RUNNABLE:DISABLE_TABLE_MARK_REGIONS_OFFLINE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:16,243 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.Procedure(966): pid=18, state=RUNNABLE:DISABLE_TABLE_MARK_REGIONS_OFFLINE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:16,243 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 14 msec 2023-07-12 15:04:16,243 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 15:04:16,244 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:16,245 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,42757,1689174248798, table=hbase:meta, region=1588230740 2023-07-12 15:04:16,247 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 1 possibly 'live' servers, and 0 'splitting'. 2023-07-12 15:04:16,248 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:16,258 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 15:04:16,258 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 15:04:16,258 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 15:04:16,259 INFO [master/jenkins-hbase5:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 15:04:16,259 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 15:04:16,260 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 15:04:16,260 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 15:04:16,262 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 15:04:16,262 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase5.apache.org,46219,1689174247629, sessionid=0x101c5ac4fea0000, setting cluster-up flag (Was=true) 2023-07-12 15:04:16,264 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 15:04:16,265 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,46219,1689174247629 2023-07-12 15:04:16,268 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 15:04:16,269 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,46219,1689174247629 2023-07-12 15:04:16,270 WARN [master/jenkins-hbase5:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/.hbase-snapshot/.tmp 2023-07-12 15:04:16,271 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:16,271 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:16,271 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:16,271 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:16,271 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase5:0, corePoolSize=10, maxPoolSize=10 2023-07-12 15:04:16,271 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:16,271 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:04:16,271 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:16,271 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174286271 2023-07-12 15:04:16,272 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 15:04:16,272 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:DISABLE_TABLE_MARK_REGIONS_OFFLINE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 15:04:16,272 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 15:04:16,272 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 15:04:16,273 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 15:04:16,273 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 15:04:16,273 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 15:04:16,273 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:16,273 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 15:04:16,273 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 15:04:16,273 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 15:04:16,274 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 15:04:16,274 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 15:04:16,274 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174256274,5,FailOnTimeoutGroup] 2023-07-12 15:04:16,274 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174256274,5,FailOnTimeoutGroup] 2023-07-12 15:04:16,274 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:16,274 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 15:04:16,274 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:16,274 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:16,274 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689174256274, completionTime=-1 2023-07-12 15:04:16,274 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 15:04:16,274 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:16,276 DEBUG [hconnection-0x243def7-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:04:16,278 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:57784, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:04:16,280 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=df807f39ee9b409bf5f8e7d69e53fb9c, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:16,281 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=077e62046fb0676013bbd7f2752a109a, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:16,281 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=737359eba122b058346d78066084ed85, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:16,281 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=7d904a9b027be183d3dcf9d6b51ad6e9, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:16,281 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=7455294f32f9f1b46d64ca42be444b4e, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,42757,1689174248798, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798, openSeqNum=2 2023-07-12 15:04:16,281 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:16,281 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174316281 2023-07-12 15:04:16,282 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174376282 2023-07-12 15:04:16,282 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 7 msec 2023-07-12 15:04:16,297 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,46219,1689174247629-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:16,297 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,46219,1689174247629-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:16,297 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,46219,1689174247629-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:16,298 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase5:46219, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:16,298 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:16,298 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 15:04:16,304 DEBUG [master/jenkins-hbase5:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 15:04:16,314 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 15:04:16,318 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 15:04:16,318 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 0.726sec 2023-07-12 15:04:16,319 INFO [master/jenkins-hbase5:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 15:04:16,319 INFO [master/jenkins-hbase5:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 15:04:16,319 INFO [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 15:04:16,319 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,46219,1689174247629-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 15:04:16,319 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=077e62046fb0676013bbd7f2752a109a, UNASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=737359eba122b058346d78066084ed85, UNASSIGN}, {pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7d904a9b027be183d3dcf9d6b51ad6e9, UNASSIGN}, {pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7455294f32f9f1b46d64ca42be444b4e, UNASSIGN}] 2023-07-12 15:04:16,319 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,46219,1689174247629-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 15:04:16,323 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7455294f32f9f1b46d64ca42be444b4e, UNASSIGN 2023-07-12 15:04:16,326 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 15:04:16,328 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 15:04:16,329 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 15:04:16,330 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=7455294f32f9f1b46d64ca42be444b4e, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:16,330 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174256330"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174256330"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174256330"}]},"ts":"1689174256330"} 2023-07-12 15:04:16,331 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testDisableTableWithFailoverAtStep3' 2023-07-12 15:04:16,332 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; CloseRegionProcedure 7455294f32f9f1b46d64ca42be444b4e, server=jenkins-hbase5.apache.org,42757,1689174248798}] 2023-07-12 15:04:16,340 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7d904a9b027be183d3dcf9d6b51ad6e9, UNASSIGN 2023-07-12 15:04:16,341 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=7d904a9b027be183d3dcf9d6b51ad6e9, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:16,341 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689174256341"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174256341"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174256341"}]},"ts":"1689174256341"} 2023-07-12 15:04:16,344 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=21, state=RUNNABLE; CloseRegionProcedure 7d904a9b027be183d3dcf9d6b51ad6e9, server=jenkins-hbase5.apache.org,42757,1689174248798}] 2023-07-12 15:04:16,350 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=737359eba122b058346d78066084ed85, UNASSIGN 2023-07-12 15:04:16,351 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=737359eba122b058346d78066084ed85, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:16,351 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689174256351"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174256351"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174256351"}]},"ts":"1689174256351"} 2023-07-12 15:04:16,363 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=20, state=RUNNABLE; CloseRegionProcedure 737359eba122b058346d78066084ed85, server=jenkins-hbase5.apache.org,42757,1689174248798}] 2023-07-12 15:04:16,372 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=077e62046fb0676013bbd7f2752a109a, UNASSIGN 2023-07-12 15:04:16,373 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=077e62046fb0676013bbd7f2752a109a, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:16,373 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174256373"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174256373"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174256373"}]},"ts":"1689174256373"} 2023-07-12 15:04:16,376 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=19, state=RUNNABLE; CloseRegionProcedure 077e62046fb0676013bbd7f2752a109a, server=jenkins-hbase5.apache.org,42757,1689174248798}] 2023-07-12 15:04:16,491 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:16,491 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 15:04:16,494 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:33930, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 15:04:16,495 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 737359eba122b058346d78066084ed85 2023-07-12 15:04:16,496 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 737359eba122b058346d78066084ed85, disabling compactions & flushes 2023-07-12 15:04:16,496 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85. 2023-07-12 15:04:16,496 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85. 2023-07-12 15:04:16,496 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85. after waiting 0 ms 2023-07-12 15:04:16,496 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85. 2023-07-12 15:04:16,501 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/737359eba122b058346d78066084ed85/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 15:04:16,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85. 2023-07-12 15:04:16,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 737359eba122b058346d78066084ed85: 2023-07-12 15:04:16,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 737359eba122b058346d78066084ed85 2023-07-12 15:04:16,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 7d904a9b027be183d3dcf9d6b51ad6e9 2023-07-12 15:04:16,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 7d904a9b027be183d3dcf9d6b51ad6e9, disabling compactions & flushes 2023-07-12 15:04:16,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9. 2023-07-12 15:04:16,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9. 2023-07-12 15:04:16,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9. after waiting 0 ms 2023-07-12 15:04:16,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9. 2023-07-12 15:04:16,506 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=737359eba122b058346d78066084ed85, regionState=CLOSED 2023-07-12 15:04:16,507 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,a,1689174252981.737359eba122b058346d78066084ed85.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689174256506"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174256506"}]},"ts":"1689174256506"} 2023-07-12 15:04:16,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/7d904a9b027be183d3dcf9d6b51ad6e9/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 15:04:16,511 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=20 2023-07-12 15:04:16,512 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=20, state=SUCCESS; CloseRegionProcedure 737359eba122b058346d78066084ed85, server=jenkins-hbase5.apache.org,42757,1689174248798 in 146 msec 2023-07-12 15:04:16,512 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9. 2023-07-12 15:04:16,512 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 7d904a9b027be183d3dcf9d6b51ad6e9: 2023-07-12 15:04:16,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 7d904a9b027be183d3dcf9d6b51ad6e9 2023-07-12 15:04:16,514 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=737359eba122b058346d78066084ed85, UNASSIGN in 193 msec 2023-07-12 15:04:16,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 077e62046fb0676013bbd7f2752a109a 2023-07-12 15:04:16,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 077e62046fb0676013bbd7f2752a109a, disabling compactions & flushes 2023-07-12 15:04:16,515 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a. 2023-07-12 15:04:16,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a. 2023-07-12 15:04:16,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a. after waiting 0 ms 2023-07-12 15:04:16,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a. 2023-07-12 15:04:16,516 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=7d904a9b027be183d3dcf9d6b51ad6e9, regionState=CLOSED 2023-07-12 15:04:16,516 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,b,1689174252981.7d904a9b027be183d3dcf9d6b51ad6e9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689174256516"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174256516"}]},"ts":"1689174256516"} 2023-07-12 15:04:16,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/077e62046fb0676013bbd7f2752a109a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 15:04:16,520 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=21 2023-07-12 15:04:16,520 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=21, state=SUCCESS; CloseRegionProcedure 7d904a9b027be183d3dcf9d6b51ad6e9, server=jenkins-hbase5.apache.org,42757,1689174248798 in 174 msec 2023-07-12 15:04:16,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a. 2023-07-12 15:04:16,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 077e62046fb0676013bbd7f2752a109a: 2023-07-12 15:04:16,522 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7d904a9b027be183d3dcf9d6b51ad6e9, UNASSIGN in 201 msec 2023-07-12 15:04:16,523 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 077e62046fb0676013bbd7f2752a109a 2023-07-12 15:04:16,523 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 7455294f32f9f1b46d64ca42be444b4e 2023-07-12 15:04:16,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 7455294f32f9f1b46d64ca42be444b4e, disabling compactions & flushes 2023-07-12 15:04:16,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e. 2023-07-12 15:04:16,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e. 2023-07-12 15:04:16,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e. after waiting 0 ms 2023-07-12 15:04:16,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e. 2023-07-12 15:04:16,524 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=077e62046fb0676013bbd7f2752a109a, regionState=CLOSED 2023-07-12 15:04:16,525 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,,1689174252981.077e62046fb0676013bbd7f2752a109a.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174256524"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174256524"}]},"ts":"1689174256524"} 2023-07-12 15:04:16,529 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=19 2023-07-12 15:04:16,529 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=19, state=SUCCESS; CloseRegionProcedure 077e62046fb0676013bbd7f2752a109a, server=jenkins-hbase5.apache.org,42757,1689174248798 in 150 msec 2023-07-12 15:04:16,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/default/testDisableTableWithFailoverAtStep3/7455294f32f9f1b46d64ca42be444b4e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 15:04:16,531 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e. 2023-07-12 15:04:16,531 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=077e62046fb0676013bbd7f2752a109a, UNASSIGN in 210 msec 2023-07-12 15:04:16,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 7455294f32f9f1b46d64ca42be444b4e: 2023-07-12 15:04:16,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 7455294f32f9f1b46d64ca42be444b4e 2023-07-12 15:04:16,534 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=7455294f32f9f1b46d64ca42be444b4e, regionState=CLOSED 2023-07-12 15:04:16,534 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,c,1689174252981.7455294f32f9f1b46d64ca42be444b4e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174256534"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174256534"}]},"ts":"1689174256534"} 2023-07-12 15:04:16,543 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-12 15:04:16,543 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; CloseRegionProcedure 7455294f32f9f1b46d64ca42be444b4e, server=jenkins-hbase5.apache.org,42757,1689174248798 in 209 msec 2023-07-12 15:04:16,548 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=18 2023-07-12 15:04:16,549 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7455294f32f9f1b46d64ca42be444b4e, UNASSIGN in 224 msec 2023-07-12 15:04:16,549 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDisableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174256549"}]},"ts":"1689174256549"} 2023-07-12 15:04:16,551 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDisableTableWithFailoverAtStep3, state=DISABLED in hbase:meta 2023-07-12 15:04:16,554 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testDisableTableWithFailoverAtStep3 to state=DISABLED 2023-07-12 15:04:16,556 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 in 2.0700 sec 2023-07-12 15:04:16,585 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 15:04:16,586 DEBUG [Listener at localhost/34167] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66452f5c to 127.0.0.1:61057 2023-07-12 15:04:16,586 DEBUG [Listener at localhost/34167] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:16,587 DEBUG [Listener at localhost/34167] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 15:04:16,587 DEBUG [Listener at localhost/34167] util.JVMClusterUtil(257): Found active master hash=335748510, stopped=false 2023-07-12 15:04:16,587 INFO [Listener at localhost/34167] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase5.apache.org,46219,1689174247629 2023-07-12 15:04:16,589 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): regionserver:42757-0x101c5ac4fea0002, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:16,589 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:16,589 INFO [Listener at localhost/34167] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:16,589 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:16,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42757-0x101c5ac4fea0002, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:16,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:16,592 DEBUG [Listener at localhost/34167] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41579235 to 127.0.0.1:61057 2023-07-12 15:04:16,592 DEBUG [Listener at localhost/34167] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:16,592 INFO [Listener at localhost/34167] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,42757,1689174248798' ***** 2023-07-12 15:04:16,592 INFO [Listener at localhost/34167] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 15:04:16,592 INFO [RS:0;jenkins-hbase5:42757] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 15:04:16,592 INFO [RS:0;jenkins-hbase5:42757] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 15:04:16,593 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 15:04:16,593 INFO [RS:0;jenkins-hbase5:42757] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 15:04:16,593 INFO [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(3305): Received CLOSE for df807f39ee9b409bf5f8e7d69e53fb9c 2023-07-12 15:04:16,594 INFO [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:16,594 DEBUG [RS:0;jenkins-hbase5:42757] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78c708ec to 127.0.0.1:61057 2023-07-12 15:04:16,594 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing df807f39ee9b409bf5f8e7d69e53fb9c, disabling compactions & flushes 2023-07-12 15:04:16,594 DEBUG [RS:0;jenkins-hbase5:42757] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:16,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c. 2023-07-12 15:04:16,594 INFO [RS:0;jenkins-hbase5:42757] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 15:04:16,594 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c. 2023-07-12 15:04:16,594 INFO [RS:0;jenkins-hbase5:42757] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 15:04:16,595 INFO [RS:0;jenkins-hbase5:42757] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 15:04:16,595 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c. after waiting 0 ms 2023-07-12 15:04:16,595 INFO [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 15:04:16,595 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c. 2023-07-12 15:04:16,595 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing df807f39ee9b409bf5f8e7d69e53fb9c 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-12 15:04:16,595 INFO [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-07-12 15:04:16,595 DEBUG [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(1478): Online Regions={df807f39ee9b409bf5f8e7d69e53fb9c=hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c., 1588230740=hbase:meta,,1.1588230740} 2023-07-12 15:04:16,595 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 15:04:16,595 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 15:04:16,595 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 15:04:16,595 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 15:04:16,595 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 15:04:16,596 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=10.44 KB heapSize=16.86 KB 2023-07-12 15:04:16,596 DEBUG [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(1504): Waiting on 1588230740, df807f39ee9b409bf5f8e7d69e53fb9c 2023-07-12 15:04:16,662 INFO [regionserver/jenkins-hbase5:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 15:04:16,797 DEBUG [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(1504): Waiting on 1588230740, df807f39ee9b409bf5f8e7d69e53fb9c 2023-07-12 15:04:16,997 DEBUG [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(1504): Waiting on 1588230740, df807f39ee9b409bf5f8e7d69e53fb9c 2023-07-12 15:04:17,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/namespace/df807f39ee9b409bf5f8e7d69e53fb9c/.tmp/info/1462a01d34fc4d52bb2f116a6ed65f47 2023-07-12 15:04:17,030 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=10.02 KB at sequenceid=31 (bloomFilter=false), to=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/.tmp/info/e8f9c813d89545d08d6b83c34da5bdb1 2023-07-12 15:04:17,042 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/namespace/df807f39ee9b409bf5f8e7d69e53fb9c/.tmp/info/1462a01d34fc4d52bb2f116a6ed65f47 as hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/namespace/df807f39ee9b409bf5f8e7d69e53fb9c/info/1462a01d34fc4d52bb2f116a6ed65f47 2023-07-12 15:04:17,051 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 15:04:17,053 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/namespace/df807f39ee9b409bf5f8e7d69e53fb9c/info/1462a01d34fc4d52bb2f116a6ed65f47, entries=2, sequenceid=6, filesize=4.8 K 2023-07-12 15:04:17,054 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for df807f39ee9b409bf5f8e7d69e53fb9c in 459ms, sequenceid=6, compaction requested=false 2023-07-12 15:04:17,062 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/namespace/df807f39ee9b409bf5f8e7d69e53fb9c/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-12 15:04:17,063 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c. 2023-07-12 15:04:17,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for df807f39ee9b409bf5f8e7d69e53fb9c: 2023-07-12 15:04:17,064 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689174251722.df807f39ee9b409bf5f8e7d69e53fb9c. 2023-07-12 15:04:17,197 DEBUG [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 15:04:17,357 INFO [regionserver/jenkins-hbase5:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-12 15:04:17,357 INFO [regionserver/jenkins-hbase5:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-12 15:04:17,397 DEBUG [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 15:04:17,453 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=429 B at sequenceid=31 (bloomFilter=false), to=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/.tmp/table/aaa0529a6b384755befa496b0fbe04a4 2023-07-12 15:04:17,461 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/.tmp/info/e8f9c813d89545d08d6b83c34da5bdb1 as hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/info/e8f9c813d89545d08d6b83c34da5bdb1 2023-07-12 15:04:17,468 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/info/e8f9c813d89545d08d6b83c34da5bdb1, entries=54, sequenceid=31, filesize=12.4 K 2023-07-12 15:04:17,469 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/.tmp/table/aaa0529a6b384755befa496b0fbe04a4 as hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/table/aaa0529a6b384755befa496b0fbe04a4 2023-07-12 15:04:17,476 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/table/aaa0529a6b384755befa496b0fbe04a4, entries=5, sequenceid=31, filesize=4.9 K 2023-07-12 15:04:17,478 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~10.44 KB/10692, heapSize ~16.58 KB/16976, currentSize=0 B/0 for 1588230740 in 883ms, sequenceid=31, compaction requested=false 2023-07-12 15:04:17,485 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/data/hbase/meta/1588230740/recovered.edits/34.seqid, newMaxSeqId=34, maxSeqId=1 2023-07-12 15:04:17,487 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 15:04:17,487 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 15:04:17,487 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 15:04:17,487 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 15:04:17,598 INFO [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,42757,1689174248798; all regions closed. 2023-07-12 15:04:17,609 DEBUG [RS:0;jenkins-hbase5:42757] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/oldWALs 2023-07-12 15:04:17,609 INFO [RS:0;jenkins-hbase5:42757] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C42757%2C1689174248798.meta:.meta(num 1689174251481) 2023-07-12 15:04:17,615 DEBUG [RS:0;jenkins-hbase5:42757] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/oldWALs 2023-07-12 15:04:17,615 INFO [RS:0;jenkins-hbase5:42757] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C42757%2C1689174248798:(num 1689174250504) 2023-07-12 15:04:17,615 DEBUG [RS:0;jenkins-hbase5:42757] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:17,616 INFO [RS:0;jenkins-hbase5:42757] regionserver.LeaseManager(133): Closed leases 2023-07-12 15:04:17,616 INFO [RS:0;jenkins-hbase5:42757] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase5:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-12 15:04:17,616 INFO [regionserver/jenkins-hbase5:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:04:17,617 INFO [RS:0;jenkins-hbase5:42757] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:42757 2023-07-12 15:04:17,622 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:04:17,622 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): regionserver:42757-0x101c5ac4fea0002, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase5.apache.org,42757,1689174248798 2023-07-12 15:04:17,622 ERROR [Listener at localhost/34167-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@52cecaa0 rejected from java.util.concurrent.ThreadPoolExecutor@56f14421[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:603) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-07-12 15:04:17,623 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): regionserver:42757-0x101c5ac4fea0002, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:04:17,624 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase5.apache.org,42757,1689174248798] 2023-07-12 15:04:17,624 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase5.apache.org,42757,1689174248798; numProcessing=1 2023-07-12 15:04:17,627 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase5.apache.org,42757,1689174248798 already deleted, retry=false 2023-07-12 15:04:17,627 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase5.apache.org,42757,1689174248798 expired; onlineServers=0 2023-07-12 15:04:17,627 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,46219,1689174247629' ***** 2023-07-12 15:04:17,627 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-12 15:04:17,627 DEBUG [M:0;jenkins-hbase5:46219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@553e17a5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:04:17,627 INFO [M:0;jenkins-hbase5:46219] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,46219,1689174247629 2023-07-12 15:04:17,627 INFO [M:0;jenkins-hbase5:46219] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,46219,1689174247629; all regions closed. 2023-07-12 15:04:17,627 DEBUG [M:0;jenkins-hbase5:46219] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:17,627 DEBUG [M:0;jenkins-hbase5:46219] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 15:04:17,627 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 15:04:17,627 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174256274] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174256274,5,FailOnTimeoutGroup] 2023-07-12 15:04:17,627 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174256274] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174256274,5,FailOnTimeoutGroup] 2023-07-12 15:04:17,627 DEBUG [M:0;jenkins-hbase5:46219] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 15:04:17,628 INFO [M:0;jenkins-hbase5:46219] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 15:04:17,628 INFO [M:0;jenkins-hbase5:46219] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 15:04:17,628 INFO [M:0;jenkins-hbase5:46219] hbase.ChoreService(369): Chore service for: master/jenkins-hbase5:0 had [] on shutdown 2023-07-12 15:04:17,628 DEBUG [M:0;jenkins-hbase5:46219] master.HMaster(1512): Stopping service threads 2023-07-12 15:04:17,628 INFO [M:0;jenkins-hbase5:46219] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 15:04:17,629 ERROR [M:0;jenkins-hbase5:46219] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:17,629 INFO [M:0;jenkins-hbase5:46219] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:17,629 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 15:04:17,629 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:17,629 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:17,629 DEBUG [M:0;jenkins-hbase5:46219] zookeeper.ZKUtil(398): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 15:04:17,629 WARN [M:0;jenkins-hbase5:46219] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 15:04:17,629 INFO [M:0;jenkins-hbase5:46219] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:17,630 INFO [M:0;jenkins-hbase5:46219] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-12 15:04:17,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:17,630 DEBUG [M:0;jenkins-hbase5:46219] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:04:17,630 INFO [M:0;jenkins-hbase5:46219] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:17,630 DEBUG [M:0;jenkins-hbase5:46219] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:17,630 DEBUG [M:0;jenkins-hbase5:46219] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:04:17,630 DEBUG [M:0;jenkins-hbase5:46219] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:17,630 INFO [M:0;jenkins-hbase5:46219] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=23.35 KB heapSize=28.81 KB 2023-07-12 15:04:17,724 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): regionserver:42757-0x101c5ac4fea0002, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:17,724 INFO [RS:0;jenkins-hbase5:42757] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,42757,1689174248798; zookeeper connection closed. 2023-07-12 15:04:17,724 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): regionserver:42757-0x101c5ac4fea0002, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:17,725 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@13447d7e] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@13447d7e 2023-07-12 15:04:17,725 INFO [Listener at localhost/34167] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 1 regionserver(s) complete 2023-07-12 15:04:18,045 INFO [M:0;jenkins-hbase5:46219] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=23.35 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/fee78755e62f49d6b32d4bd6b4e92b0a 2023-07-12 15:04:18,053 DEBUG [M:0;jenkins-hbase5:46219] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/fee78755e62f49d6b32d4bd6b4e92b0a as hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/fee78755e62f49d6b32d4bd6b4e92b0a 2023-07-12 15:04:18,060 INFO [M:0;jenkins-hbase5:46219] regionserver.HStore(1080): Added hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/fee78755e62f49d6b32d4bd6b4e92b0a, entries=9, sequenceid=190, filesize=5.4 K 2023-07-12 15:04:18,062 INFO [M:0;jenkins-hbase5:46219] regionserver.HRegion(2948): Finished flush of dataSize ~23.35 KB/23913, heapSize ~28.80 KB/29488, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 431ms, sequenceid=190, compaction requested=false 2023-07-12 15:04:18,063 INFO [M:0;jenkins-hbase5:46219] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:18,063 DEBUG [M:0;jenkins-hbase5:46219] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:18,068 INFO [M:0;jenkins-hbase5:46219] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 15:04:18,068 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:04:18,068 INFO [M:0;jenkins-hbase5:46219] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:46219 2023-07-12 15:04:18,071 DEBUG [M:0;jenkins-hbase5:46219] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase5.apache.org,46219,1689174247629 already deleted, retry=false 2023-07-12 15:04:18,173 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:18,173 INFO [M:0;jenkins-hbase5:46219] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,46219,1689174247629; zookeeper connection closed. 2023-07-12 15:04:18,173 DEBUG [Listener at localhost/34167-EventThread] zookeeper.ZKWatcher(600): master:46219-0x101c5ac4fea0000, quorum=127.0.0.1:61057, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:18,175 WARN [Listener at localhost/34167] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 15:04:18,177 INFO [Listener at localhost/34167] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 15:04:18,281 WARN [BP-10718897-172.31.10.131-1689174244642 heartbeating to localhost/127.0.0.1:46679] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 15:04:18,281 WARN [BP-10718897-172.31.10.131-1689174244642 heartbeating to localhost/127.0.0.1:46679] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-10718897-172.31.10.131-1689174244642 (Datanode Uuid 761e383d-70b7-4b64-9aed-989e1d50a70b) service to localhost/127.0.0.1:46679 2023-07-12 15:04:18,282 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/cluster_b99044a4-4030-b979-45d0-f7a87f1007da/dfs/data/data1/current/BP-10718897-172.31.10.131-1689174244642] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 15:04:18,283 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/cluster_b99044a4-4030-b979-45d0-f7a87f1007da/dfs/data/data2/current/BP-10718897-172.31.10.131-1689174244642] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 15:04:18,315 INFO [Listener at localhost/34167] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 15:04:18,424 INFO [Listener at localhost/34167] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 15:04:18,445 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-07-12 15:04:18,456 INFO [Listener at localhost/34167] hbase.ResourceChecker(175): after: master.procedure.TestMasterFailoverWithProcedures#testDisableTableWithFailover Thread=61 (was 10) Potentially hanging thread: region-location-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/34167 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcClient-timer-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase5:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: RS-EventLoopGroup-1-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (593222724) connection to localhost/127.0.0.1:46679 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: master/jenkins-hbase5:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SessionTracker java.lang.Thread.sleep(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Idle-Rpc-Conn-Sweeper-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:46679 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:327) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3693) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (593222724) connection to localhost/127.0.0.1:46679 from jenkins.hfs.0 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: ForkJoinPool-2-worker-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-1 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.0@localhost:46679 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase5:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@4eae2e8d java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: region-location-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (593222724) connection to localhost/127.0.0.1:46679 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: ForkJoinPool-2-worker-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) - Thread LEAK? -, OpenFileDescriptor=394 (was 263) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=278 (was 312), ProcessCount=161 (was 161), AvailableMemoryMB=8158 (was 8756) 2023-07-12 15:04:18,465 INFO [Listener at localhost/34167] hbase.ResourceChecker(147): before: master.procedure.TestMasterFailoverWithProcedures#testCreateWithFailover Thread=61, OpenFileDescriptor=394, MaxFileDescriptor=60000, SystemLoadAverage=278, ProcessCount=161, AvailableMemoryMB=8158 2023-07-12 15:04:18,465 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=2, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-12 15:04:18,466 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/hadoop.log.dir so I do NOT create it in target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462 2023-07-12 15:04:18,466 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/cc02cff1-5a82-d52c-812a-3d88036133c3/hadoop.tmp.dir so I do NOT create it in target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462 2023-07-12 15:04:18,466 INFO [Listener at localhost/34167] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/cluster_bdf1f7dc-b002-fb9f-7305-80631021f6f6, deleteOnExit=true 2023-07-12 15:04:18,466 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-12 15:04:18,466 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/test.cache.data in system properties and HBase conf 2023-07-12 15:04:18,466 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 15:04:18,466 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/hadoop.log.dir in system properties and HBase conf 2023-07-12 15:04:18,466 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 15:04:18,467 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 15:04:18,467 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 15:04:18,467 DEBUG [Listener at localhost/34167] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 15:04:18,467 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 15:04:18,467 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 15:04:18,467 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 15:04:18,467 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 15:04:18,468 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 15:04:18,468 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 15:04:18,468 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 15:04:18,468 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 15:04:18,468 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 15:04:18,468 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/nfs.dump.dir in system properties and HBase conf 2023-07-12 15:04:18,468 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/java.io.tmpdir in system properties and HBase conf 2023-07-12 15:04:18,468 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 15:04:18,468 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 15:04:18,468 INFO [Listener at localhost/34167] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 15:04:18,473 WARN [Listener at localhost/34167] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 15:04:18,473 WARN [Listener at localhost/34167] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 15:04:18,518 WARN [Listener at localhost/34167] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 15:04:18,521 INFO [Listener at localhost/34167] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 15:04:18,526 INFO [Listener at localhost/34167] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/java.io.tmpdir/Jetty_localhost_45119_hdfs____.xob0bi/webapp 2023-07-12 15:04:18,622 INFO [Listener at localhost/34167] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45119 2023-07-12 15:04:18,627 WARN [Listener at localhost/34167] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 15:04:18,627 WARN [Listener at localhost/34167] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 15:04:18,670 WARN [Listener at localhost/43731] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 15:04:18,688 WARN [Listener at localhost/43731] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 15:04:18,690 WARN [Listener at localhost/43731] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 15:04:18,692 INFO [Listener at localhost/43731] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 15:04:18,696 INFO [Listener at localhost/43731] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/java.io.tmpdir/Jetty_localhost_36821_datanode____.o2w92d/webapp 2023-07-12 15:04:18,789 INFO [Listener at localhost/43731] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36821 2023-07-12 15:04:18,795 WARN [Listener at localhost/46151] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 15:04:18,896 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x73f1480cd6335bfe: Processing first storage report for DS-5a4a5a67-b258-46db-b070-c67b5e3ccb45 from datanode 65d081eb-1859-4f14-9670-acdc21c4f4ba 2023-07-12 15:04:18,897 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x73f1480cd6335bfe: from storage DS-5a4a5a67-b258-46db-b070-c67b5e3ccb45 node DatanodeRegistration(127.0.0.1:40103, datanodeUuid=65d081eb-1859-4f14-9670-acdc21c4f4ba, infoPort=38003, infoSecurePort=0, ipcPort=46151, storageInfo=lv=-57;cid=testClusterID;nsid=2051891570;c=1689174258477), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 15:04:18,897 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x73f1480cd6335bfe: Processing first storage report for DS-0ec89841-9fb0-4779-9f67-f1369a705073 from datanode 65d081eb-1859-4f14-9670-acdc21c4f4ba 2023-07-12 15:04:18,897 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x73f1480cd6335bfe: from storage DS-0ec89841-9fb0-4779-9f67-f1369a705073 node DatanodeRegistration(127.0.0.1:40103, datanodeUuid=65d081eb-1859-4f14-9670-acdc21c4f4ba, infoPort=38003, infoSecurePort=0, ipcPort=46151, storageInfo=lv=-57;cid=testClusterID;nsid=2051891570;c=1689174258477), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 15:04:18,903 DEBUG [Listener at localhost/46151] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462 2023-07-12 15:04:18,906 INFO [Listener at localhost/46151] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/cluster_bdf1f7dc-b002-fb9f-7305-80631021f6f6/zookeeper_0, clientPort=54515, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/cluster_bdf1f7dc-b002-fb9f-7305-80631021f6f6/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/cluster_bdf1f7dc-b002-fb9f-7305-80631021f6f6/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 15:04:18,907 INFO [Listener at localhost/46151] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=54515 2023-07-12 15:04:18,908 INFO [Listener at localhost/46151] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:18,909 INFO [Listener at localhost/46151] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:18,924 INFO [Listener at localhost/46151] util.FSUtils(471): Created version file at hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422 with version=8 2023-07-12 15:04:18,924 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/hbase-staging 2023-07-12 15:04:18,926 INFO [Listener at localhost/46151] client.ConnectionUtils(127): master/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:04:18,927 INFO [Listener at localhost/46151] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:18,927 INFO [Listener at localhost/46151] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:18,927 INFO [Listener at localhost/46151] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:04:18,927 INFO [Listener at localhost/46151] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:18,927 INFO [Listener at localhost/46151] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:04:18,927 INFO [Listener at localhost/46151] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:04:18,928 INFO [Listener at localhost/46151] ipc.NettyRpcServer(120): Bind to /172.31.10.131:36567 2023-07-12 15:04:18,929 INFO [Listener at localhost/46151] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:18,931 INFO [Listener at localhost/46151] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:18,932 INFO [Listener at localhost/46151] zookeeper.RecoverableZooKeeper(93): Process identifier=master:36567 connecting to ZooKeeper ensemble=127.0.0.1:54515 2023-07-12 15:04:18,939 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:365670x0, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:04:18,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:36567-0x101c5ac7f350000 connected 2023-07-12 15:04:18,965 DEBUG [Listener at localhost/46151] zookeeper.ZKUtil(164): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:18,965 DEBUG [Listener at localhost/46151] zookeeper.ZKUtil(164): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:18,966 DEBUG [Listener at localhost/46151] zookeeper.ZKUtil(164): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:04:18,969 DEBUG [Listener at localhost/46151] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36567 2023-07-12 15:04:18,971 DEBUG [Listener at localhost/46151] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36567 2023-07-12 15:04:18,971 DEBUG [Listener at localhost/46151] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36567 2023-07-12 15:04:18,972 DEBUG [Listener at localhost/46151] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36567 2023-07-12 15:04:18,975 DEBUG [Listener at localhost/46151] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36567 2023-07-12 15:04:18,975 INFO [Listener at localhost/46151] master.HMaster(444): hbase.rootdir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422, hbase.cluster.distributed=false 2023-07-12 15:04:18,978 INFO [Listener at localhost/46151] client.ConnectionUtils(127): master/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:04:18,978 INFO [Listener at localhost/46151] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:18,978 INFO [Listener at localhost/46151] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:18,978 INFO [Listener at localhost/46151] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:04:18,978 INFO [Listener at localhost/46151] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:18,978 INFO [Listener at localhost/46151] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:04:18,978 INFO [Listener at localhost/46151] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:04:18,979 INFO [Listener at localhost/46151] ipc.NettyRpcServer(120): Bind to /172.31.10.131:46331 2023-07-12 15:04:18,980 INFO [Listener at localhost/46151] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:18,981 INFO [Listener at localhost/46151] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:18,982 INFO [Listener at localhost/46151] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46331 connecting to ZooKeeper ensemble=127.0.0.1:54515 2023-07-12 15:04:18,987 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:463310x0, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:04:18,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:46331-0x101c5ac7f350001 connected 2023-07-12 15:04:18,991 DEBUG [Listener at localhost/46151] zookeeper.ZKUtil(164): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:18,991 DEBUG [Listener at localhost/46151] zookeeper.ZKUtil(164): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:18,991 DEBUG [Listener at localhost/46151] zookeeper.ZKUtil(164): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:04:18,995 DEBUG [Listener at localhost/46151] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46331 2023-07-12 15:04:18,995 DEBUG [Listener at localhost/46151] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46331 2023-07-12 15:04:18,999 DEBUG [Listener at localhost/46151] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46331 2023-07-12 15:04:18,999 DEBUG [Listener at localhost/46151] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46331 2023-07-12 15:04:18,999 DEBUG [Listener at localhost/46151] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46331 2023-07-12 15:04:19,000 INFO [Listener at localhost/46151] master.HMaster(444): hbase.rootdir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422, hbase.cluster.distributed=false 2023-07-12 15:04:19,014 INFO [Listener at localhost/46151] client.ConnectionUtils(127): regionserver/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:04:19,014 INFO [Listener at localhost/46151] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:19,014 INFO [Listener at localhost/46151] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:19,014 INFO [Listener at localhost/46151] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:04:19,014 INFO [Listener at localhost/46151] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:19,014 INFO [Listener at localhost/46151] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:04:19,015 INFO [Listener at localhost/46151] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:04:19,016 INFO [Listener at localhost/46151] ipc.NettyRpcServer(120): Bind to /172.31.10.131:43707 2023-07-12 15:04:19,016 INFO [Listener at localhost/46151] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 15:04:19,017 DEBUG [Listener at localhost/46151] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 15:04:19,017 INFO [Listener at localhost/46151] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:19,019 INFO [Listener at localhost/46151] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:19,020 INFO [Listener at localhost/46151] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43707 connecting to ZooKeeper ensemble=127.0.0.1:54515 2023-07-12 15:04:19,023 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): regionserver:437070x0, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:04:19,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:43707-0x101c5ac7f350002 connected 2023-07-12 15:04:19,024 DEBUG [Listener at localhost/46151] zookeeper.ZKUtil(164): regionserver:43707-0x101c5ac7f350002, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:19,024 DEBUG [Listener at localhost/46151] zookeeper.ZKUtil(164): regionserver:43707-0x101c5ac7f350002, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:19,025 DEBUG [Listener at localhost/46151] zookeeper.ZKUtil(164): regionserver:43707-0x101c5ac7f350002, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:04:19,025 DEBUG [Listener at localhost/46151] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43707 2023-07-12 15:04:19,025 DEBUG [Listener at localhost/46151] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43707 2023-07-12 15:04:19,026 DEBUG [Listener at localhost/46151] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43707 2023-07-12 15:04:19,031 DEBUG [Listener at localhost/46151] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43707 2023-07-12 15:04:19,031 DEBUG [Listener at localhost/46151] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43707 2023-07-12 15:04:19,033 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase5.apache.org,36567,1689174258925 2023-07-12 15:04:19,033 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase5.apache.org,46331,1689174258977 2023-07-12 15:04:19,035 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:19,035 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:19,035 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase5.apache.org,36567,1689174258925 2023-07-12 15:04:19,035 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase5.apache.org,46331,1689174258977 2023-07-12 15:04:19,040 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:19,040 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:19,040 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:19,040 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x101c5ac7f350002, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:19,040 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:19,041 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:19,042 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:19,042 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase5.apache.org,36567,1689174258925 from backup master directory 2023-07-12 15:04:19,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:19,043 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase5.apache.org,36567,1689174258925; waiting to become the next active master 2023-07-12 15:04:19,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:19,044 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:19,044 WARN [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:04:19,044 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase5.apache.org,36567,1689174258925 2023-07-12 15:04:19,044 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase5.apache.org,36567,1689174258925 2023-07-12 15:04:19,044 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:19,088 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/hbase.id with ID: 066f4562-8b53-4583-bde9-10e047223cbf 2023-07-12 15:04:19,101 INFO [master/jenkins-hbase5:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:19,104 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:19,104 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:19,516 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0a8fedae to 127.0.0.1:54515 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:19,520 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a163472, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:19,521 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:19,521 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 15:04:19,522 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:19,523 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store-tmp 2023-07-12 15:04:19,934 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:19,934 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:04:19,934 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:19,934 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:19,934 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:04:19,934 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:19,934 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:19,934 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:19,935 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/WALs/jenkins-hbase5.apache.org,36567,1689174258925 2023-07-12 15:04:19,939 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C36567%2C1689174258925, suffix=, logDir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/WALs/jenkins-hbase5.apache.org,36567,1689174258925, archiveDir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/oldWALs, maxLogs=10 2023-07-12 15:04:19,962 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40103,DS-5a4a5a67-b258-46db-b070-c67b5e3ccb45,DISK] 2023-07-12 15:04:19,971 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/WALs/jenkins-hbase5.apache.org,36567,1689174258925/jenkins-hbase5.apache.org%2C36567%2C1689174258925.1689174259940 2023-07-12 15:04:19,973 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40103,DS-5a4a5a67-b258-46db-b070-c67b5e3ccb45,DISK]] 2023-07-12 15:04:19,974 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:19,974 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:19,974 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:19,974 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:19,979 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:19,982 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 15:04:19,982 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 15:04:19,983 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:19,984 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:19,985 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:19,990 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:20,004 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:20,005 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70668048, jitterRate=0.05303597450256348}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:04:20,005 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:20,006 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 15:04:20,008 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:20,008 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:20,008 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:20,009 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:20,009 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-07-12 15:04:20,009 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 15:04:20,020 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 15:04:20,021 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 15:04:20,032 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 15:04:20,032 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 15:04:20,033 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 15:04:20,033 INFO [master/jenkins-hbase5:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 15:04:20,033 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 15:04:20,035 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:20,035 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:20,036 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 15:04:20,036 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 15:04:20,037 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 15:04:20,038 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:20,038 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:20,038 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x101c5ac7f350002, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:20,038 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:20,038 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:20,039 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase5.apache.org,36567,1689174258925, sessionid=0x101c5ac7f350000, setting cluster-up flag (Was=false) 2023-07-12 15:04:20,041 INFO [M:1;jenkins-hbase5:46331] regionserver.HRegionServer(951): ClusterId : 066f4562-8b53-4583-bde9-10e047223cbf 2023-07-12 15:04:20,044 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:20,044 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:20,048 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 15:04:20,049 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,36567,1689174258925 2023-07-12 15:04:20,056 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:20,056 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:20,061 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 15:04:20,062 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,36567,1689174258925 2023-07-12 15:04:20,063 WARN [master/jenkins-hbase5:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.hbase-snapshot/.tmp 2023-07-12 15:04:20,065 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:20,065 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:20,065 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:20,065 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:20,065 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:20,065 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase5:0, corePoolSize=10, maxPoolSize=10 2023-07-12 15:04:20,065 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:20,065 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:04:20,065 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:20,066 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174290066 2023-07-12 15:04:20,066 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 15:04:20,066 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 15:04:20,066 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 15:04:20,067 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 15:04:20,067 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 15:04:20,067 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 15:04:20,067 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:20,067 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 15:04:20,067 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 15:04:20,067 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 15:04:20,068 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:20,068 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 15:04:20,068 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 15:04:20,068 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 15:04:20,069 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:20,071 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174260068,5,FailOnTimeoutGroup] 2023-07-12 15:04:20,071 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174260071,5,FailOnTimeoutGroup] 2023-07-12 15:04:20,072 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:20,072 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 15:04:20,072 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:20,072 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:20,135 INFO [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(951): ClusterId : 066f4562-8b53-4583-bde9-10e047223cbf 2023-07-12 15:04:20,135 DEBUG [RS:0;jenkins-hbase5:43707] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 15:04:20,138 DEBUG [RS:0;jenkins-hbase5:43707] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 15:04:20,138 DEBUG [RS:0;jenkins-hbase5:43707] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 15:04:20,140 DEBUG [RS:0;jenkins-hbase5:43707] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 15:04:20,141 DEBUG [RS:0;jenkins-hbase5:43707] zookeeper.ReadOnlyZKClient(139): Connect 0x75eb6097 to 127.0.0.1:54515 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:20,145 DEBUG [RS:0;jenkins-hbase5:43707] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f130be7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:20,145 DEBUG [RS:0;jenkins-hbase5:43707] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a118a61, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:04:20,154 DEBUG [RS:0;jenkins-hbase5:43707] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase5:43707 2023-07-12 15:04:20,154 INFO [RS:0;jenkins-hbase5:43707] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 15:04:20,154 INFO [RS:0;jenkins-hbase5:43707] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 15:04:20,154 DEBUG [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 15:04:20,155 INFO [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase5.apache.org,36567,1689174258925 with isa=jenkins-hbase5.apache.org/172.31.10.131:43707, startcode=1689174259013 2023-07-12 15:04:20,155 DEBUG [RS:0;jenkins-hbase5:43707] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 15:04:20,158 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:59795, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 15:04:20,159 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36567] master.ServerManager(394): Registering regionserver=jenkins-hbase5.apache.org,43707,1689174259013 2023-07-12 15:04:20,160 DEBUG [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422 2023-07-12 15:04:20,160 DEBUG [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:43731 2023-07-12 15:04:20,160 DEBUG [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 15:04:20,163 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:04:20,163 DEBUG [RS:0;jenkins-hbase5:43707] zookeeper.ZKUtil(162): regionserver:43707-0x101c5ac7f350002, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,43707,1689174259013 2023-07-12 15:04:20,163 WARN [RS:0;jenkins-hbase5:43707] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:04:20,163 INFO [RS:0;jenkins-hbase5:43707] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:20,164 DEBUG [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(1948): logDir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/WALs/jenkins-hbase5.apache.org,43707,1689174259013 2023-07-12 15:04:20,164 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase5.apache.org,43707,1689174259013] 2023-07-12 15:04:20,167 DEBUG [RS:0;jenkins-hbase5:43707] zookeeper.ZKUtil(162): regionserver:43707-0x101c5ac7f350002, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,43707,1689174259013 2023-07-12 15:04:20,168 DEBUG [RS:0;jenkins-hbase5:43707] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 15:04:20,169 INFO [RS:0;jenkins-hbase5:43707] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 15:04:20,171 INFO [RS:0;jenkins-hbase5:43707] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 15:04:20,171 INFO [RS:0;jenkins-hbase5:43707] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:04:20,171 INFO [RS:0;jenkins-hbase5:43707] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:20,171 INFO [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 15:04:20,173 INFO [RS:0;jenkins-hbase5:43707] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:20,173 DEBUG [RS:0;jenkins-hbase5:43707] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:20,173 DEBUG [RS:0;jenkins-hbase5:43707] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:20,173 DEBUG [RS:0;jenkins-hbase5:43707] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:20,173 DEBUG [RS:0;jenkins-hbase5:43707] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:20,173 DEBUG [RS:0;jenkins-hbase5:43707] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:20,173 DEBUG [RS:0;jenkins-hbase5:43707] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:04:20,173 DEBUG [RS:0;jenkins-hbase5:43707] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:20,173 DEBUG [RS:0;jenkins-hbase5:43707] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:20,173 DEBUG [RS:0;jenkins-hbase5:43707] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:20,173 DEBUG [RS:0;jenkins-hbase5:43707] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:20,174 INFO [RS:0;jenkins-hbase5:43707] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:20,174 INFO [RS:0;jenkins-hbase5:43707] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:20,174 INFO [RS:0;jenkins-hbase5:43707] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:20,186 INFO [RS:0;jenkins-hbase5:43707] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 15:04:20,186 INFO [RS:0;jenkins-hbase5:43707] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,43707,1689174259013-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:20,197 INFO [RS:0;jenkins-hbase5:43707] regionserver.Replication(203): jenkins-hbase5.apache.org,43707,1689174259013 started 2023-07-12 15:04:20,197 INFO [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(1637): Serving as jenkins-hbase5.apache.org,43707,1689174259013, RpcServer on jenkins-hbase5.apache.org/172.31.10.131:43707, sessionid=0x101c5ac7f350002 2023-07-12 15:04:20,198 DEBUG [RS:0;jenkins-hbase5:43707] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 15:04:20,198 DEBUG [RS:0;jenkins-hbase5:43707] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase5.apache.org,43707,1689174259013 2023-07-12 15:04:20,198 DEBUG [RS:0;jenkins-hbase5:43707] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,43707,1689174259013' 2023-07-12 15:04:20,198 DEBUG [RS:0;jenkins-hbase5:43707] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:04:20,198 DEBUG [RS:0;jenkins-hbase5:43707] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:04:20,198 DEBUG [RS:0;jenkins-hbase5:43707] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 15:04:20,199 DEBUG [RS:0;jenkins-hbase5:43707] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 15:04:20,199 DEBUG [RS:0;jenkins-hbase5:43707] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase5.apache.org,43707,1689174259013 2023-07-12 15:04:20,199 DEBUG [RS:0;jenkins-hbase5:43707] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,43707,1689174259013' 2023-07-12 15:04:20,199 DEBUG [RS:0;jenkins-hbase5:43707] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 15:04:20,199 DEBUG [RS:0;jenkins-hbase5:43707] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 15:04:20,199 DEBUG [RS:0;jenkins-hbase5:43707] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 15:04:20,199 INFO [RS:0;jenkins-hbase5:43707] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 15:04:20,199 INFO [RS:0;jenkins-hbase5:43707] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 15:04:20,302 INFO [RS:0;jenkins-hbase5:43707] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C43707%2C1689174259013, suffix=, logDir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/WALs/jenkins-hbase5.apache.org,43707,1689174259013, archiveDir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/oldWALs, maxLogs=32 2023-07-12 15:04:20,320 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40103,DS-5a4a5a67-b258-46db-b070-c67b5e3ccb45,DISK] 2023-07-12 15:04:20,322 INFO [RS:0;jenkins-hbase5:43707] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/WALs/jenkins-hbase5.apache.org,43707,1689174259013/jenkins-hbase5.apache.org%2C43707%2C1689174259013.1689174260304 2023-07-12 15:04:20,322 DEBUG [RS:0;jenkins-hbase5:43707] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40103,DS-5a4a5a67-b258-46db-b070-c67b5e3ccb45,DISK]] 2023-07-12 15:04:20,481 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:20,481 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:20,481 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422 2023-07-12 15:04:20,891 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:20,893 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 15:04:20,895 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/info 2023-07-12 15:04:20,896 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 15:04:20,896 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:20,897 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 15:04:20,898 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:04:20,898 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 15:04:20,899 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:20,899 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 15:04:20,901 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/table 2023-07-12 15:04:20,901 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 15:04:20,902 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:20,903 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740 2023-07-12 15:04:20,904 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740 2023-07-12 15:04:20,906 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 15:04:20,908 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 15:04:20,910 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:20,910 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73912964, jitterRate=0.10138899087905884}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 15:04:20,911 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 15:04:20,911 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 15:04:20,911 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 15:04:20,911 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 15:04:20,911 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 15:04:20,911 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 15:04:20,911 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 15:04:20,911 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 15:04:20,912 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:20,912 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 15:04:20,913 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 15:04:20,914 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 15:04:20,915 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 15:04:21,065 DEBUG [jenkins-hbase5:36567] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 15:04:21,066 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,43707,1689174259013, state=OPENING 2023-07-12 15:04:21,068 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 15:04:21,069 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:21,069 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:21,069 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,43707,1689174259013}] 2023-07-12 15:04:21,069 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:04:21,069 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:04:21,222 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase5.apache.org,43707,1689174259013 2023-07-12 15:04:21,223 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 15:04:21,226 INFO [RS-EventLoopGroup-8-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:50824, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 15:04:21,230 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 15:04:21,231 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:21,232 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C43707%2C1689174259013.meta, suffix=.meta, logDir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/WALs/jenkins-hbase5.apache.org,43707,1689174259013, archiveDir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/oldWALs, maxLogs=32 2023-07-12 15:04:21,248 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40103,DS-5a4a5a67-b258-46db-b070-c67b5e3ccb45,DISK] 2023-07-12 15:04:21,250 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/WALs/jenkins-hbase5.apache.org,43707,1689174259013/jenkins-hbase5.apache.org%2C43707%2C1689174259013.meta.1689174261234.meta 2023-07-12 15:04:21,251 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40103,DS-5a4a5a67-b258-46db-b070-c67b5e3ccb45,DISK]] 2023-07-12 15:04:21,251 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:21,251 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 15:04:21,251 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 15:04:21,252 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 15:04:21,252 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 15:04:21,252 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:21,252 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 15:04:21,252 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 15:04:21,253 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 15:04:21,255 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/info 2023-07-12 15:04:21,255 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/info 2023-07-12 15:04:21,255 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 15:04:21,256 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:21,256 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 15:04:21,257 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:04:21,257 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:04:21,257 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 15:04:21,258 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:21,258 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 15:04:21,259 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/table 2023-07-12 15:04:21,259 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/table 2023-07-12 15:04:21,259 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 15:04:21,260 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:21,261 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740 2023-07-12 15:04:21,263 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740 2023-07-12 15:04:21,265 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 15:04:21,267 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 15:04:21,268 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65232875, jitterRate=-0.02795441448688507}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 15:04:21,268 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 15:04:21,269 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689174261222 2023-07-12 15:04:21,273 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 15:04:21,273 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 15:04:21,273 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,43707,1689174259013, state=OPEN 2023-07-12 15:04:21,279 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 15:04:21,279 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 15:04:21,279 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:04:21,279 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:04:21,281 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 15:04:21,282 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,43707,1689174259013 in 210 msec 2023-07-12 15:04:21,284 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 15:04:21,284 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 369 msec 2023-07-12 15:04:21,286 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.2210 sec 2023-07-12 15:04:21,286 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689174261286, completionTime=-1 2023-07-12 15:04:21,287 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 15:04:21,287 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:21,289 DEBUG [hconnection-0x1876a2ac-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:04:21,292 INFO [RS-EventLoopGroup-8-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:50836, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:04:21,294 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:21,294 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174321294 2023-07-12 15:04:21,294 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174381294 2023-07-12 15:04:21,294 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 7 msec 2023-07-12 15:04:21,301 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,36567,1689174258925-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:21,301 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,36567,1689174258925-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:21,301 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,36567,1689174258925-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:21,301 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase5:36567, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:21,301 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:21,302 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 15:04:21,302 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:21,303 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:21,303 DEBUG [master/jenkins-hbase5:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 15:04:21,305 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:04:21,306 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:04:21,308 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.tmp/data/hbase/namespace/9f1f58001670ba8a9f6e5d540454c39c 2023-07-12 15:04:21,309 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.tmp/data/hbase/namespace/9f1f58001670ba8a9f6e5d540454c39c empty. 2023-07-12 15:04:21,309 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.tmp/data/hbase/namespace/9f1f58001670ba8a9f6e5d540454c39c 2023-07-12 15:04:21,309 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 15:04:21,322 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:21,323 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9f1f58001670ba8a9f6e5d540454c39c, NAME => 'hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.tmp 2023-07-12 15:04:21,331 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:21,331 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 9f1f58001670ba8a9f6e5d540454c39c, disabling compactions & flushes 2023-07-12 15:04:21,331 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c. 2023-07-12 15:04:21,331 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c. 2023-07-12 15:04:21,331 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c. after waiting 0 ms 2023-07-12 15:04:21,331 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c. 2023-07-12 15:04:21,331 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c. 2023-07-12 15:04:21,331 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 9f1f58001670ba8a9f6e5d540454c39c: 2023-07-12 15:04:21,334 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:04:21,336 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689174261336"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174261336"}]},"ts":"1689174261336"} 2023-07-12 15:04:21,338 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 15:04:21,339 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:04:21,339 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174261339"}]},"ts":"1689174261339"} 2023-07-12 15:04:21,341 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 15:04:21,347 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=9f1f58001670ba8a9f6e5d540454c39c, ASSIGN}] 2023-07-12 15:04:21,350 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=9f1f58001670ba8a9f6e5d540454c39c, ASSIGN 2023-07-12 15:04:21,350 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=9f1f58001670ba8a9f6e5d540454c39c, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,43707,1689174259013; forceNewPlan=false, retain=false 2023-07-12 15:04:21,501 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=9f1f58001670ba8a9f6e5d540454c39c, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,43707,1689174259013 2023-07-12 15:04:21,502 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689174261501"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174261501"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174261501"}]},"ts":"1689174261501"} 2023-07-12 15:04:21,508 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 9f1f58001670ba8a9f6e5d540454c39c, server=jenkins-hbase5.apache.org,43707,1689174259013}] 2023-07-12 15:04:21,664 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c. 2023-07-12 15:04:21,665 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9f1f58001670ba8a9f6e5d540454c39c, NAME => 'hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:21,666 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 9f1f58001670ba8a9f6e5d540454c39c 2023-07-12 15:04:21,666 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:21,666 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 9f1f58001670ba8a9f6e5d540454c39c 2023-07-12 15:04:21,666 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 9f1f58001670ba8a9f6e5d540454c39c 2023-07-12 15:04:21,667 INFO [StoreOpener-9f1f58001670ba8a9f6e5d540454c39c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 9f1f58001670ba8a9f6e5d540454c39c 2023-07-12 15:04:21,669 DEBUG [StoreOpener-9f1f58001670ba8a9f6e5d540454c39c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/namespace/9f1f58001670ba8a9f6e5d540454c39c/info 2023-07-12 15:04:21,669 DEBUG [StoreOpener-9f1f58001670ba8a9f6e5d540454c39c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/namespace/9f1f58001670ba8a9f6e5d540454c39c/info 2023-07-12 15:04:21,669 INFO [StoreOpener-9f1f58001670ba8a9f6e5d540454c39c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9f1f58001670ba8a9f6e5d540454c39c columnFamilyName info 2023-07-12 15:04:21,670 INFO [StoreOpener-9f1f58001670ba8a9f6e5d540454c39c-1] regionserver.HStore(310): Store=9f1f58001670ba8a9f6e5d540454c39c/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:21,671 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/namespace/9f1f58001670ba8a9f6e5d540454c39c 2023-07-12 15:04:21,672 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/namespace/9f1f58001670ba8a9f6e5d540454c39c 2023-07-12 15:04:21,676 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 9f1f58001670ba8a9f6e5d540454c39c 2023-07-12 15:04:21,678 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/namespace/9f1f58001670ba8a9f6e5d540454c39c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:21,678 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 9f1f58001670ba8a9f6e5d540454c39c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75209216, jitterRate=0.12070465087890625}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:04:21,679 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 9f1f58001670ba8a9f6e5d540454c39c: 2023-07-12 15:04:21,680 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c., pid=6, masterSystemTime=1689174261660 2023-07-12 15:04:21,682 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c. 2023-07-12 15:04:21,682 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c. 2023-07-12 15:04:21,683 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=9f1f58001670ba8a9f6e5d540454c39c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,43707,1689174259013 2023-07-12 15:04:21,683 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689174261682"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174261682"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174261682"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174261682"}]},"ts":"1689174261682"} 2023-07-12 15:04:21,687 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 15:04:21,687 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 9f1f58001670ba8a9f6e5d540454c39c, server=jenkins-hbase5.apache.org,43707,1689174259013 in 177 msec 2023-07-12 15:04:21,690 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 15:04:21,690 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=9f1f58001670ba8a9f6e5d540454c39c, ASSIGN in 340 msec 2023-07-12 15:04:21,691 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 15:04:21,691 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174261691"}]},"ts":"1689174261691"} 2023-07-12 15:04:21,692 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 15:04:21,695 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 15:04:21,697 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 393 msec 2023-07-12 15:04:21,705 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 15:04:21,706 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:21,706 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:04:21,706 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:21,710 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:21,718 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:04:21,726 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 16 msec 2023-07-12 15:04:21,732 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:21,739 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:04:21,743 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 11 msec 2023-07-12 15:04:21,757 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 15:04:21,759 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 15:04:21,759 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.714sec 2023-07-12 15:04:21,759 INFO [master/jenkins-hbase5:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 15:04:21,759 INFO [master/jenkins-hbase5:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 15:04:21,759 INFO [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 15:04:21,759 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,36567,1689174258925-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 15:04:21,760 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,36567,1689174258925-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 15:04:21,762 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 15:04:21,835 DEBUG [Listener at localhost/46151] zookeeper.ReadOnlyZKClient(139): Connect 0x57372864 to 127.0.0.1:54515 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:21,840 DEBUG [Listener at localhost/46151] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65a9954, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:21,841 DEBUG [hconnection-0x451deb93-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:04:21,844 INFO [RS-EventLoopGroup-8-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:50846, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:04:21,846 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase5.apache.org,36567,1689174258925 2023-07-12 15:04:21,846 WARN [Listener at localhost/46151] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-12 15:04:21,846 WARN [Listener at localhost/46151] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-07-12 15:04:21,847 DEBUG [Listener at localhost/46151] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 15:04:21,848 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:04:21,849 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:21,849 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 15:04:21,849 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:21,849 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:22,009 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-12 15:04:22,097 INFO [Listener at localhost/46151] procedure.TestMasterFailoverWithProcedures(306): Restart 0 exec state: pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 15:04:22,098 INFO [Listener at localhost/46151] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:04:22,098 INFO [Listener at localhost/46151] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:22,098 INFO [Listener at localhost/46151] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:22,099 ERROR [Listener at localhost/46151] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-6,5,PEWorkerGroup] 2023-07-12 15:04:22,099 INFO [Listener at localhost/46151] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:04:22,099 INFO [Listener at localhost/46151] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:22,099 INFO [Listener at localhost/46151] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:22,099 INFO [Listener at localhost/46151] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:22,100 INFO [Listener at localhost/46151] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:22,101 DEBUG [Listener at localhost/46151] store.ProcedureTree(114): Procedure Procedure(pid=9, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.CreateTableProcedure) stack ids=[] 2023-07-12 15:04:22,101 DEBUG [Listener at localhost/46151] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:22,102 DEBUG [Listener at localhost/46151] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:22,102 DEBUG [Listener at localhost/46151] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:22,102 DEBUG [Listener at localhost/46151] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:22,103 DEBUG [Listener at localhost/46151] procedure2.ProcedureExecutor(468): Loading pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 15:04:22,103 DEBUG [Listener at localhost/46151] procedure2.Procedure(966): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCreateWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:22,103 INFO [Listener at localhost/46151] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 15:04:22,104 DEBUG [Listener at localhost/46151] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:22,104 DEBUG [Listener at localhost/46151] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,43707,1689174259013, table=hbase:meta, region=1588230740 2023-07-12 15:04:22,105 INFO [Listener at localhost/46151] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174292105 2023-07-12 15:04:22,105 DEBUG [Listener at localhost/46151] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:22,105 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 15:04:22,108 DEBUG [Listener at localhost/46151] assignment.RegionStateStore(147): Load hbase:meta entry region=9f1f58001670ba8a9f6e5d540454c39c, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,43707,1689174259013, regionLocation=jenkins-hbase5.apache.org,43707,1689174259013, openSeqNum=2 2023-07-12 15:04:22,108 INFO [Listener at localhost/46151] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:22,108 INFO [Listener at localhost/46151] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174322108 2023-07-12 15:04:22,108 INFO [Listener at localhost/46151] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174382108 2023-07-12 15:04:22,108 INFO [Listener at localhost/46151] assignment.AssignmentManager(1536): Joined the cluster in 2 msec 2023-07-12 15:04:22,108 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:04:22,108 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:04:22,110 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:04:22,112 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.tmp/data/default/testCreateWithFailoverAtStep3/83da04dad54d09c08dd7adad02269e13 2023-07-12 15:04:22,113 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.tmp/data/default/testCreateWithFailoverAtStep3/83da04dad54d09c08dd7adad02269e13 empty. 2023-07-12 15:04:22,114 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.tmp/data/default/testCreateWithFailoverAtStep3/83da04dad54d09c08dd7adad02269e13 2023-07-12 15:04:22,114 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCreateWithFailoverAtStep3 regions 2023-07-12 15:04:22,528 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.tmp/data/default/testCreateWithFailoverAtStep3/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:22,529 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(7675): creating {ENCODED => 83da04dad54d09c08dd7adad02269e13, NAME => 'testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testCreateWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.tmp 2023-07-12 15:04:22,939 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(866): Instantiated testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:22,940 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1604): Closing 83da04dad54d09c08dd7adad02269e13, disabling compactions & flushes 2023-07-12 15:04:22,940 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1626): Closing region testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. 2023-07-12 15:04:22,940 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. 2023-07-12 15:04:22,940 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1714): Acquired close lock on testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. after waiting 0 ms 2023-07-12 15:04:22,940 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1724): Updates disabled for region testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. 2023-07-12 15:04:22,940 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1838): Closed testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. 2023-07-12 15:04:22,940 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1558): Region close journal for 83da04dad54d09c08dd7adad02269e13: 2023-07-12 15:04:22,942 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:22,942 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 15:04:22,942 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:22,942 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:23,109 INFO [Listener at localhost/46151] procedure.TestMasterFailoverWithProcedures(306): Restart 1 exec state: pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 15:04:23,109 INFO [Listener at localhost/46151] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:04:23,109 INFO [Listener at localhost/46151] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:23,109 INFO [Listener at localhost/46151] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:23,109 ERROR [Listener at localhost/46151] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-7,5,PEWorkerGroup] 2023-07-12 15:04:23,109 INFO [Listener at localhost/46151] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:04:23,109 INFO [Listener at localhost/46151] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:23,109 INFO [Listener at localhost/46151] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:23,109 INFO [Listener at localhost/46151] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:23,110 INFO [Listener at localhost/46151] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:23,111 DEBUG [Listener at localhost/46151] store.ProcedureTree(114): Procedure Procedure(pid=9, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.CreateTableProcedure) stack ids=[0] 2023-07-12 15:04:23,111 DEBUG [Listener at localhost/46151] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:23,112 DEBUG [Listener at localhost/46151] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:23,112 DEBUG [Listener at localhost/46151] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:23,112 DEBUG [Listener at localhost/46151] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:23,112 DEBUG [Listener at localhost/46151] procedure2.ProcedureExecutor(468): Loading pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 15:04:23,113 DEBUG [Listener at localhost/46151] procedure2.Procedure(966): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testCreateWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:23,113 INFO [Listener at localhost/46151] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 15:04:23,113 DEBUG [Listener at localhost/46151] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:23,114 DEBUG [Listener at localhost/46151] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,43707,1689174259013, table=hbase:meta, region=1588230740 2023-07-12 15:04:23,114 INFO [Listener at localhost/46151] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174293114 2023-07-12 15:04:23,115 DEBUG [Listener at localhost/46151] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:23,115 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 15:04:23,117 DEBUG [Listener at localhost/46151] assignment.RegionStateStore(147): Load hbase:meta entry region=9f1f58001670ba8a9f6e5d540454c39c, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,43707,1689174259013, regionLocation=jenkins-hbase5.apache.org,43707,1689174259013, openSeqNum=2 2023-07-12 15:04:23,118 INFO [Listener at localhost/46151] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:23,118 INFO [Listener at localhost/46151] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174323118 2023-07-12 15:04:23,118 INFO [Listener at localhost/46151] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174383118 2023-07-12 15:04:23,118 INFO [Listener at localhost/46151] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 15:04:23,118 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:04:23,122 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.tmp/data/default/testCreateWithFailoverAtStep3/83da04dad54d09c08dd7adad02269e13 2023-07-12 15:04:23,123 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.tmp/data/default/testCreateWithFailoverAtStep3/83da04dad54d09c08dd7adad02269e13 empty. 2023-07-12 15:04:23,124 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.tmp/data/default/testCreateWithFailoverAtStep3/83da04dad54d09c08dd7adad02269e13 2023-07-12 15:04:23,124 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCreateWithFailoverAtStep3 regions 2023-07-12 15:04:23,535 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.tmp/data/default/testCreateWithFailoverAtStep3/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:23,537 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(7675): creating {ENCODED => 83da04dad54d09c08dd7adad02269e13, NAME => 'testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testCreateWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.tmp 2023-07-12 15:04:23,946 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(866): Instantiated testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:23,946 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1604): Closing 83da04dad54d09c08dd7adad02269e13, disabling compactions & flushes 2023-07-12 15:04:23,946 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1626): Closing region testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. 2023-07-12 15:04:23,946 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. 2023-07-12 15:04:23,946 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1714): Acquired close lock on testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. after waiting 0 ms 2023-07-12 15:04:23,946 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1724): Updates disabled for region testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. 2023-07-12 15:04:23,946 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1838): Closed testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. 2023-07-12 15:04:23,946 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1558): Region close journal for 83da04dad54d09c08dd7adad02269e13: 2023-07-12 15:04:23,948 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:04:23,950 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:04:23,952 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689174263951"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174263951"}]},"ts":"1689174263951"} 2023-07-12 15:04:23,954 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 15:04:23,954 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:23,954 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 15:04:23,954 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:23,954 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:24,118 INFO [Listener at localhost/46151] procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state: pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 15:04:24,118 INFO [Listener at localhost/46151] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:04:24,118 INFO [Listener at localhost/46151] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:24,119 INFO [Listener at localhost/46151] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:24,119 ERROR [Listener at localhost/46151] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-07-12 15:04:24,119 INFO [Listener at localhost/46151] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:04:24,119 INFO [Listener at localhost/46151] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:24,119 INFO [Listener at localhost/46151] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:24,119 INFO [Listener at localhost/46151] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:24,120 INFO [Listener at localhost/46151] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:24,121 DEBUG [Listener at localhost/46151] store.ProcedureTree(114): Procedure Procedure(pid=9, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.CreateTableProcedure) stack ids=[0, 1] 2023-07-12 15:04:24,121 DEBUG [Listener at localhost/46151] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:24,121 DEBUG [Listener at localhost/46151] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:24,122 DEBUG [Listener at localhost/46151] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:24,122 DEBUG [Listener at localhost/46151] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:24,122 DEBUG [Listener at localhost/46151] procedure2.ProcedureExecutor(468): Loading pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 15:04:24,122 DEBUG [Listener at localhost/46151] procedure2.Procedure(966): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testCreateWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:24,122 INFO [Listener at localhost/46151] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 15:04:24,123 DEBUG [Listener at localhost/46151] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:24,124 DEBUG [Listener at localhost/46151] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,43707,1689174259013, table=hbase:meta, region=1588230740 2023-07-12 15:04:24,124 INFO [Listener at localhost/46151] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174294124 2023-07-12 15:04:24,124 DEBUG [Listener at localhost/46151] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:24,125 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 15:04:24,127 DEBUG [Listener at localhost/46151] assignment.RegionStateStore(147): Load hbase:meta entry region=9f1f58001670ba8a9f6e5d540454c39c, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,43707,1689174259013, regionLocation=jenkins-hbase5.apache.org,43707,1689174259013, openSeqNum=2 2023-07-12 15:04:24,127 DEBUG [Listener at localhost/46151] assignment.RegionStateStore(147): Load hbase:meta entry region=83da04dad54d09c08dd7adad02269e13, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-12 15:04:24,128 INFO [Listener at localhost/46151] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:24,128 INFO [Listener at localhost/46151] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174324128 2023-07-12 15:04:24,128 INFO [Listener at localhost/46151] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174384128 2023-07-12 15:04:24,128 INFO [Listener at localhost/46151] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 15:04:24,128 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:04:24,129 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689174264129"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174264129"}]},"ts":"1689174264129"} 2023-07-12 15:04:24,131 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 15:04:24,131 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:04:24,132 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:04:24,133 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCreateWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174264133"}]},"ts":"1689174264133"} 2023-07-12 15:04:24,134 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCreateWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-12 15:04:24,139 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCreateWithFailoverAtStep3, region=83da04dad54d09c08dd7adad02269e13, ASSIGN}] 2023-07-12 15:04:24,139 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:24,139 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=9, state=WAITING:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 15:04:24,139 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:24,139 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=9, state=WAITING:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:24,378 INFO [Listener at localhost/46151] procedure.TestMasterFailoverWithProcedures(313): Trigger master failover 2023-07-12 15:04:24,378 INFO [Listener at localhost/46151] hbase.MiniHBaseCluster(607): Aborting jenkins-hbase5.apache.org,36567,1689174258925 2023-07-12 15:04:24,378 ERROR [Listener at localhost/46151] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 15:04:24,378 ERROR [Listener at localhost/46151] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase5.apache.org,36567,1689174258925: Aborting for tests ***** java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:608) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:411) at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.masterFailover(MasterProcedureTestingUtility.java:135) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testRecoveryAndDoubleExecution(TestMasterFailoverWithProcedures.java:314) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testCreateWithFailoverAtStep(TestMasterFailoverWithProcedures.java:121) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testCreateWithFailover(TestMasterFailoverWithProcedures.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:04:24,379 INFO [Listener at localhost/46151] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,36567,1689174258925' ***** 2023-07-12 15:04:24,379 INFO [Listener at localhost/46151] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost/46151 2023-07-12 15:04:24,379 DEBUG [M:0;jenkins-hbase5:36567] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c4831d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:04:24,380 INFO [M:0;jenkins-hbase5:36567] regionserver.HRegionServer(1141): aborting server jenkins-hbase5.apache.org,36567,1689174258925 2023-07-12 15:04:24,380 DEBUG [M:0;jenkins-hbase5:36567] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a8fedae to 127.0.0.1:54515 2023-07-12 15:04:24,380 DEBUG [M:0;jenkins-hbase5:36567] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:24,380 INFO [M:0;jenkins-hbase5:36567] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,36567,1689174258925; all regions closed. 2023-07-12 15:04:24,380 DEBUG [M:0;jenkins-hbase5:36567] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:24,380 DEBUG [M:0;jenkins-hbase5:36567] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 15:04:24,380 DEBUG [M:0;jenkins-hbase5:36567] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 15:04:24,380 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 15:04:24,380 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174260071] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174260071,5,FailOnTimeoutGroup] 2023-07-12 15:04:24,380 INFO [M:0;jenkins-hbase5:36567] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 15:04:24,380 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174260068] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174260068,5,FailOnTimeoutGroup] 2023-07-12 15:04:24,381 INFO [M:0;jenkins-hbase5:36567] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 15:04:24,382 INFO [M:0;jenkins-hbase5:36567] hbase.ChoreService(369): Chore service for: master/jenkins-hbase5:0 had [] on shutdown 2023-07-12 15:04:24,382 DEBUG [M:0;jenkins-hbase5:36567] master.HMaster(1512): Stopping service threads 2023-07-12 15:04:24,382 INFO [M:0;jenkins-hbase5:36567] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 15:04:24,383 ERROR [M:0;jenkins-hbase5:36567] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:24,383 INFO [M:0;jenkins-hbase5:36567] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 15:04:24,383 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 15:04:24,387 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:24,387 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x101c5ac7f350002, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:24,388 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:24,387 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:24,388 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:24,388 DEBUG [M:0;jenkins-hbase5:36567] zookeeper.ZKUtil(398): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 15:04:24,388 WARN [M:0;jenkins-hbase5:36567] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 15:04:24,388 INFO [M:0;jenkins-hbase5:36567] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:24,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43707-0x101c5ac7f350002, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:24,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:24,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:24,388 INFO [M:0;jenkins-hbase5:36567] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 15:04:24,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:24,389 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-12 15:04:24,391 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:24,391 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x101c5ac7f350002, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:24,391 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:24,391 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:24,391 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:24,392 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:24,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:24,392 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase5.apache.org,46331,1689174258977 from backup master directory 2023-07-12 15:04:24,393 DEBUG [M:0;jenkins-hbase5:36567] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:04:24,393 INFO [M:0;jenkins-hbase5:36567] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:24,393 DEBUG [M:0;jenkins-hbase5:36567] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:24,393 DEBUG [M:0;jenkins-hbase5:36567] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:04:24,393 DEBUG [M:0;jenkins-hbase5:36567] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:24,393 ERROR [M:0;jenkins-hbase5:36567] regionserver.HRegion(1825): Memstore data size is 29049 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:24,393 INFO [M:0;jenkins-hbase5:36567] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:24,393 DEBUG [M:0;jenkins-hbase5:36567] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:24,393 INFO [M:0;jenkins-hbase5:36567] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 15:04:24,393 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:04:24,394 INFO [M:0;jenkins-hbase5:36567] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:36567 2023-07-12 15:04:24,394 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase5.apache.org,46331,1689174258977 2023-07-12 15:04:24,396 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:24,397 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:24,397 WARN [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:04:24,397 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase5.apache.org,46331,1689174258977 2023-07-12 15:04:24,398 DEBUG [M:0;jenkins-hbase5:36567] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase5.apache.org,36567,1689174258925 already deleted, retry=false 2023-07-12 15:04:24,411 INFO [master/jenkins-hbase5:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:24,430 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x568e0112 to 127.0.0.1:54515 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:24,433 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@430ea0ff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:24,433 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:24,434 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 15:04:24,434 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:24,440 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/WALs/jenkins-hbase5.apache.org,36567,1689174258925 to hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/WALs/jenkins-hbase5.apache.org,36567,1689174258925-dead as it is dead 2023-07-12 15:04:24,440 INFO [master/jenkins-hbase5:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/WALs/jenkins-hbase5.apache.org,36567,1689174258925-dead/jenkins-hbase5.apache.org%2C36567%2C1689174258925.1689174259940 2023-07-12 15:04:24,441 INFO [master/jenkins-hbase5:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/WALs/jenkins-hbase5.apache.org,36567,1689174258925-dead/jenkins-hbase5.apache.org%2C36567%2C1689174258925.1689174259940 after 0ms 2023-07-12 15:04:24,442 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/WALs/jenkins-hbase5.apache.org,36567,1689174258925-dead/jenkins-hbase5.apache.org%2C36567%2C1689174258925.1689174259940 to hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C36567%2C1689174258925.1689174259940 2023-07-12 15:04:24,442 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/WALs/jenkins-hbase5.apache.org,36567,1689174258925-dead 2023-07-12 15:04:24,442 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/WALs/jenkins-hbase5.apache.org,46331,1689174258977 2023-07-12 15:04:24,445 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C46331%2C1689174258977, suffix=, logDir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/WALs/jenkins-hbase5.apache.org,46331,1689174258977, archiveDir=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/oldWALs, maxLogs=10 2023-07-12 15:04:24,459 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40103,DS-5a4a5a67-b258-46db-b070-c67b5e3ccb45,DISK] 2023-07-12 15:04:24,462 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/WALs/jenkins-hbase5.apache.org,46331,1689174258977/jenkins-hbase5.apache.org%2C46331%2C1689174258977.1689174264445 2023-07-12 15:04:24,463 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40103,DS-5a4a5a67-b258-46db-b070-c67b5e3ccb45,DISK]] 2023-07-12 15:04:24,463 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:24,463 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:24,463 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:24,463 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:24,465 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:24,466 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 15:04:24,466 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 15:04:24,467 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:24,468 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-12 15:04:24,468 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C36567%2C1689174258925.1689174259940 2023-07-12 15:04:24,476 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 76, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=70, path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C36567%2C1689174258925.1689174259940 2023-07-12 15:04:24,476 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=28.37 KB heapSize=34.23 KB 2023-07-12 15:04:24,489 WARN [RS-EventLoopGroup-8-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase5.apache.org/172.31.10.131:36567 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:36567 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:04:24,490 DEBUG [RS-EventLoopGroup-8-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase5.apache.org/172.31.10.131:36567 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:36567 2023-07-12 15:04:24,582 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:24,583 INFO [M:0;jenkins-hbase5:36567] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,36567,1689174258925; zookeeper connection closed. 2023-07-12 15:04:24,583 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:36567-0x101c5ac7f350000, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:24,591 DEBUG [RS:0;jenkins-hbase5:43707] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 15:04:24,594 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:40185, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 15:04:24,594 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46331] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:24,697 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46331] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:24,800 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46331] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:24,891 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=28.37 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/76613340ac2648cb850ecdfa45ed8855 2023-07-12 15:04:24,899 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/76613340ac2648cb850ecdfa45ed8855 as hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/76613340ac2648cb850ecdfa45ed8855 2023-07-12 15:04:24,903 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46331] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:24,906 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/76613340ac2648cb850ecdfa45ed8855, entries=76, sequenceid=70, filesize=33.2 K 2023-07-12 15:04:24,906 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~28.37 KB/29049, heapSize ~34.21 KB/35032, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 430ms, sequenceid=70, compaction requested=false; wal=null 2023-07-12 15:04:24,907 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C36567%2C1689174258925.1689174259940 2023-07-12 15:04:24,912 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:24,914 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/70.seqid, newMaxSeqId=70, maxSeqId=1 2023-07-12 15:04:24,915 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=71; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59320567, jitterRate=-0.11605466902256012}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:04:24,915 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:24,915 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 15:04:24,916 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:24,916 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:24,916 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:24,917 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:24,921 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] store.ProcedureTree(114): Procedure Procedure(pid=9, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.CreateTableProcedure) stack ids=[0, 1, 2] 2023-07-12 15:04:24,921 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:24,922 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:24,922 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:24,922 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:24,923 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(468): Loading pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 15:04:24,923 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.Procedure(966): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testCreateWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:24,923 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-07-12 15:04:24,923 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 15:04:24,924 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:24,925 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,43707,1689174259013, table=hbase:meta, region=1588230740 2023-07-12 15:04:24,926 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 1 possibly 'live' servers, and 0 'splitting'. 2023-07-12 15:04:24,927 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase5.apache.org,43707,1689174259013 2023-07-12 15:04:24,938 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 15:04:24,938 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 15:04:24,939 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 15:04:24,939 INFO [master/jenkins-hbase5:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 15:04:24,939 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 15:04:24,941 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 15:04:24,941 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 15:04:24,942 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 15:04:24,942 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase5.apache.org,46331,1689174258977, sessionid=0x101c5ac7f350001, setting cluster-up flag (Was=true) 2023-07-12 15:04:24,944 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 15:04:24,944 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,46331,1689174258977 2023-07-12 15:04:24,947 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 15:04:24,948 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,46331,1689174258977 2023-07-12 15:04:24,949 WARN [master/jenkins-hbase5:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/.hbase-snapshot/.tmp 2023-07-12 15:04:24,950 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:24,950 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:24,950 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:24,950 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:24,950 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase5:0, corePoolSize=10, maxPoolSize=10 2023-07-12 15:04:24,950 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:24,950 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:04:24,950 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:24,950 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174294950 2023-07-12 15:04:24,951 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 15:04:24,951 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 15:04:24,951 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 15:04:24,951 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 15:04:24,952 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 15:04:24,952 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 15:04:24,952 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 15:04:24,952 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:24,952 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 15:04:24,952 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 15:04:24,952 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 15:04:24,953 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 15:04:24,953 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 15:04:24,953 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174264953,5,FailOnTimeoutGroup] 2023-07-12 15:04:24,953 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174264953,5,FailOnTimeoutGroup] 2023-07-12 15:04:24,953 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:24,953 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 15:04:24,953 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:24,953 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:24,953 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689174264953, completionTime=-1 2023-07-12 15:04:24,954 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 15:04:24,954 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:24,955 DEBUG [hconnection-0x3eaf84d8-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:04:24,957 INFO [RS-EventLoopGroup-8-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:50854, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:04:24,959 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=9f1f58001670ba8a9f6e5d540454c39c, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,43707,1689174259013, regionLocation=jenkins-hbase5.apache.org,43707,1689174259013, openSeqNum=2 2023-07-12 15:04:24,960 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=83da04dad54d09c08dd7adad02269e13, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-12 15:04:24,960 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:24,960 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174324960 2023-07-12 15:04:24,960 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174384960 2023-07-12 15:04:24,960 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-07-12 15:04:24,974 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,46331,1689174258977-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:24,974 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,46331,1689174258977-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:24,974 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,46331,1689174258977-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:24,974 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase5:46331, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:24,974 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:24,975 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 15:04:24,977 DEBUG [master/jenkins-hbase5:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 15:04:24,985 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 15:04:24,989 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 15:04:24,989 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 0.592sec 2023-07-12 15:04:24,989 INFO [master/jenkins-hbase5:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 15:04:24,989 INFO [master/jenkins-hbase5:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 15:04:24,989 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:04:24,989 INFO [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 15:04:24,989 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,46331,1689174258977-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 15:04:24,989 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCreateWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174264989"}]},"ts":"1689174264989"} 2023-07-12 15:04:24,989 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,46331,1689174258977-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 15:04:24,992 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCreateWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-12 15:04:24,992 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 15:04:24,995 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCreateWithFailoverAtStep3, region=83da04dad54d09c08dd7adad02269e13, ASSIGN}] 2023-07-12 15:04:24,998 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCreateWithFailoverAtStep3, region=83da04dad54d09c08dd7adad02269e13, ASSIGN 2023-07-12 15:04:24,999 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCreateWithFailoverAtStep3, region=83da04dad54d09c08dd7adad02269e13, ASSIGN; state=CLOSED, location=jenkins-hbase5.apache.org,43707,1689174259013; forceNewPlan=false, retain=false 2023-07-12 15:04:25,150 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=83da04dad54d09c08dd7adad02269e13, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,43707,1689174259013 2023-07-12 15:04:25,150 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689174265150"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174265150"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174265150"}]},"ts":"1689174265150"} 2023-07-12 15:04:25,153 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 83da04dad54d09c08dd7adad02269e13, server=jenkins-hbase5.apache.org,43707,1689174259013}] 2023-07-12 15:04:25,306 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase5.apache.org,43707,1689174259013 2023-07-12 15:04:25,306 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 15:04:25,308 INFO [RS-EventLoopGroup-8-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:50862, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 15:04:25,313 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. 2023-07-12 15:04:25,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 83da04dad54d09c08dd7adad02269e13, NAME => 'testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:25,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCreateWithFailoverAtStep3 83da04dad54d09c08dd7adad02269e13 2023-07-12 15:04:25,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:25,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 83da04dad54d09c08dd7adad02269e13 2023-07-12 15:04:25,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 83da04dad54d09c08dd7adad02269e13 2023-07-12 15:04:25,315 INFO [StoreOpener-83da04dad54d09c08dd7adad02269e13-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 83da04dad54d09c08dd7adad02269e13 2023-07-12 15:04:25,316 DEBUG [StoreOpener-83da04dad54d09c08dd7adad02269e13-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/default/testCreateWithFailoverAtStep3/83da04dad54d09c08dd7adad02269e13/f1 2023-07-12 15:04:25,316 DEBUG [StoreOpener-83da04dad54d09c08dd7adad02269e13-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/default/testCreateWithFailoverAtStep3/83da04dad54d09c08dd7adad02269e13/f1 2023-07-12 15:04:25,317 INFO [StoreOpener-83da04dad54d09c08dd7adad02269e13-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 83da04dad54d09c08dd7adad02269e13 columnFamilyName f1 2023-07-12 15:04:25,317 INFO [StoreOpener-83da04dad54d09c08dd7adad02269e13-1] regionserver.HStore(310): Store=83da04dad54d09c08dd7adad02269e13/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:25,318 INFO [StoreOpener-83da04dad54d09c08dd7adad02269e13-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 83da04dad54d09c08dd7adad02269e13 2023-07-12 15:04:25,319 DEBUG [StoreOpener-83da04dad54d09c08dd7adad02269e13-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/default/testCreateWithFailoverAtStep3/83da04dad54d09c08dd7adad02269e13/f2 2023-07-12 15:04:25,319 DEBUG [StoreOpener-83da04dad54d09c08dd7adad02269e13-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/default/testCreateWithFailoverAtStep3/83da04dad54d09c08dd7adad02269e13/f2 2023-07-12 15:04:25,320 INFO [StoreOpener-83da04dad54d09c08dd7adad02269e13-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 83da04dad54d09c08dd7adad02269e13 columnFamilyName f2 2023-07-12 15:04:25,321 INFO [StoreOpener-83da04dad54d09c08dd7adad02269e13-1] regionserver.HStore(310): Store=83da04dad54d09c08dd7adad02269e13/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:25,321 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/default/testCreateWithFailoverAtStep3/83da04dad54d09c08dd7adad02269e13 2023-07-12 15:04:25,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/default/testCreateWithFailoverAtStep3/83da04dad54d09c08dd7adad02269e13 2023-07-12 15:04:25,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testCreateWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:04:25,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 83da04dad54d09c08dd7adad02269e13 2023-07-12 15:04:25,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/default/testCreateWithFailoverAtStep3/83da04dad54d09c08dd7adad02269e13/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:25,328 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 83da04dad54d09c08dd7adad02269e13; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63795592, jitterRate=-0.04937160015106201}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:04:25,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 83da04dad54d09c08dd7adad02269e13: 2023-07-12 15:04:25,329 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13., pid=11, masterSystemTime=1689174265306 2023-07-12 15:04:25,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. 2023-07-12 15:04:25,331 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. 2023-07-12 15:04:25,332 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=83da04dad54d09c08dd7adad02269e13, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,43707,1689174259013 2023-07-12 15:04:25,332 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689174265332"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174265332"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174265332"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174265332"}]},"ts":"1689174265332"} 2023-07-12 15:04:25,336 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-12 15:04:25,336 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 83da04dad54d09c08dd7adad02269e13, server=jenkins-hbase5.apache.org,43707,1689174259013 in 181 msec 2023-07-12 15:04:25,338 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-12 15:04:25,338 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testCreateWithFailoverAtStep3, region=83da04dad54d09c08dd7adad02269e13, ASSIGN in 341 msec 2023-07-12 15:04:25,339 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 15:04:25,339 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCreateWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174265339"}]},"ts":"1689174265339"} 2023-07-12 15:04:25,340 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCreateWithFailoverAtStep3, state=ENABLED in hbase:meta 2023-07-12 15:04:25,343 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 15:04:25,344 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testCreateWithFailoverAtStep3 in 3.4970 sec 2023-07-12 15:04:25,389 DEBUG [Listener at localhost/46151] util.CommonFSUtils(682): File system contents for path hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/default/testCreateWithFailoverAtStep3 2023-07-12 15:04:25,389 DEBUG [Listener at localhost/46151] util.CommonFSUtils(699): |-.tabledesc/ 2023-07-12 15:04:25,390 DEBUG [Listener at localhost/46151] util.CommonFSUtils(702): |----.tableinfo.0000000001 2023-07-12 15:04:25,390 DEBUG [Listener at localhost/46151] util.CommonFSUtils(699): |-.tmp/ 2023-07-12 15:04:25,390 DEBUG [Listener at localhost/46151] util.CommonFSUtils(699): |-83da04dad54d09c08dd7adad02269e13/ 2023-07-12 15:04:25,391 DEBUG [Listener at localhost/46151] util.CommonFSUtils(702): |----.regioninfo 2023-07-12 15:04:25,391 DEBUG [Listener at localhost/46151] util.CommonFSUtils(699): |----f1/ 2023-07-12 15:04:25,391 DEBUG [Listener at localhost/46151] util.CommonFSUtils(699): |----f2/ 2023-07-12 15:04:25,392 DEBUG [Listener at localhost/46151] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-12 15:04:25,392 DEBUG [Listener at localhost/46151] util.CommonFSUtils(702): |-------1.seqid 2023-07-12 15:04:25,398 DEBUG [Listener at localhost/46151] procedure.MasterProcedureTestingUtility(212): Table directory layout is as expected. 2023-07-12 15:04:25,403 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 15:04:25,404 DEBUG [Listener at localhost/46151] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x57372864 to 127.0.0.1:54515 2023-07-12 15:04:25,404 DEBUG [Listener at localhost/46151] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:25,405 DEBUG [Listener at localhost/46151] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 15:04:25,405 DEBUG [Listener at localhost/46151] util.JVMClusterUtil(257): Found active master hash=1807740401, stopped=false 2023-07-12 15:04:25,405 INFO [Listener at localhost/46151] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase5.apache.org,46331,1689174258977 2023-07-12 15:04:25,407 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:25,407 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x101c5ac7f350002, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:25,407 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:25,407 INFO [Listener at localhost/46151] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:25,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:25,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43707-0x101c5ac7f350002, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:25,409 DEBUG [Listener at localhost/46151] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x568e0112 to 127.0.0.1:54515 2023-07-12 15:04:25,409 DEBUG [Listener at localhost/46151] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:25,409 INFO [Listener at localhost/46151] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,43707,1689174259013' ***** 2023-07-12 15:04:25,409 INFO [Listener at localhost/46151] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 15:04:25,409 INFO [RS:0;jenkins-hbase5:43707] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 15:04:25,410 INFO [RS:0;jenkins-hbase5:43707] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 15:04:25,410 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 15:04:25,410 INFO [RS:0;jenkins-hbase5:43707] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 15:04:25,410 INFO [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(3305): Received CLOSE for 83da04dad54d09c08dd7adad02269e13 2023-07-12 15:04:25,410 INFO [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(3305): Received CLOSE for 9f1f58001670ba8a9f6e5d540454c39c 2023-07-12 15:04:25,410 INFO [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,43707,1689174259013 2023-07-12 15:04:25,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 83da04dad54d09c08dd7adad02269e13, disabling compactions & flushes 2023-07-12 15:04:25,411 DEBUG [RS:0;jenkins-hbase5:43707] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x75eb6097 to 127.0.0.1:54515 2023-07-12 15:04:25,411 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. 2023-07-12 15:04:25,411 DEBUG [RS:0;jenkins-hbase5:43707] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:25,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. 2023-07-12 15:04:25,411 INFO [RS:0;jenkins-hbase5:43707] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 15:04:25,411 INFO [RS:0;jenkins-hbase5:43707] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 15:04:25,411 INFO [RS:0;jenkins-hbase5:43707] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 15:04:25,411 INFO [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 15:04:25,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. after waiting 0 ms 2023-07-12 15:04:25,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. 2023-07-12 15:04:25,412 INFO [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-12 15:04:25,412 DEBUG [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(1478): Online Regions={83da04dad54d09c08dd7adad02269e13=testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13., 9f1f58001670ba8a9f6e5d540454c39c=hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c., 1588230740=hbase:meta,,1.1588230740} 2023-07-12 15:04:25,412 DEBUG [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(1504): Waiting on 1588230740, 83da04dad54d09c08dd7adad02269e13, 9f1f58001670ba8a9f6e5d540454c39c 2023-07-12 15:04:25,412 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 15:04:25,412 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 15:04:25,412 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 15:04:25,412 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 15:04:25,412 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 15:04:25,413 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=3.09 KB heapSize=5.82 KB 2023-07-12 15:04:25,417 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/default/testCreateWithFailoverAtStep3/83da04dad54d09c08dd7adad02269e13/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 15:04:25,418 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. 2023-07-12 15:04:25,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 83da04dad54d09c08dd7adad02269e13: 2023-07-12 15:04:25,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testCreateWithFailoverAtStep3,,1689174261846.83da04dad54d09c08dd7adad02269e13. 2023-07-12 15:04:25,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 9f1f58001670ba8a9f6e5d540454c39c, disabling compactions & flushes 2023-07-12 15:04:25,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c. 2023-07-12 15:04:25,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c. 2023-07-12 15:04:25,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c. after waiting 0 ms 2023-07-12 15:04:25,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c. 2023-07-12 15:04:25,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 9f1f58001670ba8a9f6e5d540454c39c 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-12 15:04:25,427 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.82 KB at sequenceid=16 (bloomFilter=false), to=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/.tmp/info/9dff670333c84f729dc32d64b59c2e70 2023-07-12 15:04:25,431 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/namespace/9f1f58001670ba8a9f6e5d540454c39c/.tmp/info/699604b4e2e24608934efef8b5d50a26 2023-07-12 15:04:25,439 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/namespace/9f1f58001670ba8a9f6e5d540454c39c/.tmp/info/699604b4e2e24608934efef8b5d50a26 as hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/namespace/9f1f58001670ba8a9f6e5d540454c39c/info/699604b4e2e24608934efef8b5d50a26 2023-07-12 15:04:25,447 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=277 B at sequenceid=16 (bloomFilter=false), to=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/.tmp/table/67ed480cb29e46459b7c4484e559e7eb 2023-07-12 15:04:25,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/namespace/9f1f58001670ba8a9f6e5d540454c39c/info/699604b4e2e24608934efef8b5d50a26, entries=2, sequenceid=6, filesize=4.8 K 2023-07-12 15:04:25,449 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 9f1f58001670ba8a9f6e5d540454c39c in 30ms, sequenceid=6, compaction requested=false 2023-07-12 15:04:25,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 15:04:25,456 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/.tmp/info/9dff670333c84f729dc32d64b59c2e70 as hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/info/9dff670333c84f729dc32d64b59c2e70 2023-07-12 15:04:25,462 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/info/9dff670333c84f729dc32d64b59c2e70, entries=20, sequenceid=16, filesize=7.2 K 2023-07-12 15:04:25,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/namespace/9f1f58001670ba8a9f6e5d540454c39c/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-12 15:04:25,463 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/.tmp/table/67ed480cb29e46459b7c4484e559e7eb as hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/table/67ed480cb29e46459b7c4484e559e7eb 2023-07-12 15:04:25,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c. 2023-07-12 15:04:25,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 9f1f58001670ba8a9f6e5d540454c39c: 2023-07-12 15:04:25,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689174261302.9f1f58001670ba8a9f6e5d540454c39c. 2023-07-12 15:04:25,470 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/table/67ed480cb29e46459b7c4484e559e7eb, entries=5, sequenceid=16, filesize=4.9 K 2023-07-12 15:04:25,471 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.09 KB/3163, heapSize ~5.54 KB/5672, currentSize=0 B/0 for 1588230740 in 59ms, sequenceid=16, compaction requested=false 2023-07-12 15:04:25,471 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 15:04:25,477 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/data/hbase/meta/1588230740/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=1 2023-07-12 15:04:25,478 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 15:04:25,478 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 15:04:25,478 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 15:04:25,478 INFO [regionserver/jenkins-hbase5:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 15:04:25,478 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 15:04:25,612 INFO [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,43707,1689174259013; all regions closed. 2023-07-12 15:04:25,618 DEBUG [RS:0;jenkins-hbase5:43707] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/oldWALs 2023-07-12 15:04:25,618 INFO [RS:0;jenkins-hbase5:43707] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C43707%2C1689174259013.meta:.meta(num 1689174261234) 2023-07-12 15:04:25,623 DEBUG [RS:0;jenkins-hbase5:43707] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/oldWALs 2023-07-12 15:04:25,623 INFO [RS:0;jenkins-hbase5:43707] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C43707%2C1689174259013:(num 1689174260304) 2023-07-12 15:04:25,623 DEBUG [RS:0;jenkins-hbase5:43707] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:25,623 INFO [RS:0;jenkins-hbase5:43707] regionserver.LeaseManager(133): Closed leases 2023-07-12 15:04:25,623 INFO [RS:0;jenkins-hbase5:43707] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase5:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-12 15:04:25,623 INFO [regionserver/jenkins-hbase5:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:04:25,624 INFO [RS:0;jenkins-hbase5:43707] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:43707 2023-07-12 15:04:25,626 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x101c5ac7f350002, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase5.apache.org,43707,1689174259013 2023-07-12 15:04:25,626 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:04:25,626 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x101c5ac7f350002, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:04:25,628 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase5.apache.org,43707,1689174259013] 2023-07-12 15:04:25,629 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase5.apache.org,43707,1689174259013; numProcessing=1 2023-07-12 15:04:25,631 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase5.apache.org,43707,1689174259013 already deleted, retry=false 2023-07-12 15:04:25,631 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase5.apache.org,43707,1689174259013 expired; onlineServers=0 2023-07-12 15:04:25,631 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,46331,1689174258977' ***** 2023-07-12 15:04:25,631 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-12 15:04:25,631 DEBUG [M:1;jenkins-hbase5:46331] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f80645e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:04:25,632 INFO [M:1;jenkins-hbase5:46331] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,46331,1689174258977 2023-07-12 15:04:25,632 INFO [M:1;jenkins-hbase5:46331] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,46331,1689174258977; all regions closed. 2023-07-12 15:04:25,632 DEBUG [M:1;jenkins-hbase5:46331] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:25,632 DEBUG [M:1;jenkins-hbase5:46331] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 15:04:25,632 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 15:04:25,632 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174264953] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174264953,5,FailOnTimeoutGroup] 2023-07-12 15:04:25,632 DEBUG [M:1;jenkins-hbase5:46331] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 15:04:25,632 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174264953] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174264953,5,FailOnTimeoutGroup] 2023-07-12 15:04:25,632 INFO [M:1;jenkins-hbase5:46331] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 15:04:25,633 INFO [M:1;jenkins-hbase5:46331] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 15:04:25,633 INFO [M:1;jenkins-hbase5:46331] hbase.ChoreService(369): Chore service for: master/jenkins-hbase5:0 had [] on shutdown 2023-07-12 15:04:25,633 DEBUG [M:1;jenkins-hbase5:46331] master.HMaster(1512): Stopping service threads 2023-07-12 15:04:25,633 INFO [M:1;jenkins-hbase5:46331] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 15:04:25,633 ERROR [M:1;jenkins-hbase5:46331] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:25,633 INFO [M:1;jenkins-hbase5:46331] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:25,633 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 15:04:25,633 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:25,634 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:25,634 DEBUG [M:1;jenkins-hbase5:46331] zookeeper.ZKUtil(398): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 15:04:25,634 WARN [M:1;jenkins-hbase5:46331] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 15:04:25,634 INFO [M:1;jenkins-hbase5:46331] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:25,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:25,634 INFO [M:1;jenkins-hbase5:46331] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-12 15:04:25,635 DEBUG [M:1;jenkins-hbase5:46331] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:04:25,635 INFO [M:1;jenkins-hbase5:46331] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:25,635 DEBUG [M:1;jenkins-hbase5:46331] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:25,635 DEBUG [M:1;jenkins-hbase5:46331] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:04:25,635 DEBUG [M:1;jenkins-hbase5:46331] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:25,635 INFO [M:1;jenkins-hbase5:46331] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=11.98 KB heapSize=14.01 KB 2023-07-12 15:04:25,807 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x101c5ac7f350002, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:25,807 INFO [RS:0;jenkins-hbase5:43707] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,43707,1689174259013; zookeeper connection closed. 2023-07-12 15:04:25,807 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): regionserver:43707-0x101c5ac7f350002, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:25,808 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3775931e] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3775931e 2023-07-12 15:04:25,808 INFO [Listener at localhost/46151] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 1 regionserver(s) complete 2023-07-12 15:04:26,045 INFO [M:1;jenkins-hbase5:46331] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.98 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c21eceedb2b04d5585ab7105bb175798 2023-07-12 15:04:26,052 DEBUG [M:1;jenkins-hbase5:46331] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c21eceedb2b04d5585ab7105bb175798 as hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c21eceedb2b04d5585ab7105bb175798 2023-07-12 15:04:26,058 INFO [M:1;jenkins-hbase5:46331] regionserver.HStore(1080): Added hdfs://localhost:43731/user/jenkins/test-data/91d1b967-ef11-7dbd-0c5b-2608e9d38422/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c21eceedb2b04d5585ab7105bb175798, entries=3, sequenceid=92, filesize=5.7 K 2023-07-12 15:04:26,059 INFO [M:1;jenkins-hbase5:46331] regionserver.HRegion(2948): Finished flush of dataSize ~11.98 KB/12264, heapSize ~13.99 KB/14328, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 424ms, sequenceid=92, compaction requested=false 2023-07-12 15:04:26,061 INFO [M:1;jenkins-hbase5:46331] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:26,061 DEBUG [M:1;jenkins-hbase5:46331] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:26,064 INFO [M:1;jenkins-hbase5:46331] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 15:04:26,064 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:04:26,064 INFO [M:1;jenkins-hbase5:46331] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:46331 2023-07-12 15:04:26,066 DEBUG [M:1;jenkins-hbase5:46331] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase5.apache.org,46331,1689174258977 already deleted, retry=false 2023-07-12 15:04:26,172 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:26,172 INFO [M:1;jenkins-hbase5:46331] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,46331,1689174258977; zookeeper connection closed. 2023-07-12 15:04:26,172 DEBUG [Listener at localhost/46151-EventThread] zookeeper.ZKWatcher(600): master:46331-0x101c5ac7f350001, quorum=127.0.0.1:54515, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:26,173 WARN [Listener at localhost/46151] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 15:04:26,176 INFO [Listener at localhost/46151] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 15:04:26,280 WARN [BP-1585212865-172.31.10.131-1689174258477 heartbeating to localhost/127.0.0.1:43731] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 15:04:26,280 WARN [BP-1585212865-172.31.10.131-1689174258477 heartbeating to localhost/127.0.0.1:43731] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1585212865-172.31.10.131-1689174258477 (Datanode Uuid 65d081eb-1859-4f14-9670-acdc21c4f4ba) service to localhost/127.0.0.1:43731 2023-07-12 15:04:26,281 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/cluster_bdf1f7dc-b002-fb9f-7305-80631021f6f6/dfs/data/data1/current/BP-1585212865-172.31.10.131-1689174258477] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 15:04:26,281 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/cluster_bdf1f7dc-b002-fb9f-7305-80631021f6f6/dfs/data/data2/current/BP-1585212865-172.31.10.131-1689174258477] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 15:04:26,293 INFO [Listener at localhost/46151] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 15:04:26,403 INFO [Listener at localhost/46151] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 15:04:26,413 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-07-12 15:04:26,423 INFO [Listener at localhost/46151] hbase.ResourceChecker(175): after: master.procedure.TestMasterFailoverWithProcedures#testCreateWithFailover Thread=86 (was 61) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'DataNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: Listener at localhost/46151 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: IPC Client (593222724) connection to localhost/127.0.0.1:43731 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: LeaseRenewer:jenkins@localhost:43731 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (593222724) connection to localhost/127.0.0.1:43731 from jenkins.hfs.1 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.1@localhost:43731 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (593222724) connection to localhost/127.0.0.1:43731 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-4-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=421 (was 394) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=250 (was 278), ProcessCount=161 (was 161), AvailableMemoryMB=8071 (was 8158) 2023-07-12 15:04:26,431 INFO [Listener at localhost/46151] hbase.ResourceChecker(147): before: master.procedure.TestMasterFailoverWithProcedures#testDeleteWithFailover Thread=86, OpenFileDescriptor=421, MaxFileDescriptor=60000, SystemLoadAverage=250, ProcessCount=161, AvailableMemoryMB=8071 2023-07-12 15:04:26,431 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=2, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-12 15:04:26,432 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/hadoop.log.dir so I do NOT create it in target/test-data/8514f246-f21b-9108-ea99-c782f0acda35 2023-07-12 15:04:26,432 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acfb4cee-eab5-5a01-52b2-48700c814462/hadoop.tmp.dir so I do NOT create it in target/test-data/8514f246-f21b-9108-ea99-c782f0acda35 2023-07-12 15:04:26,432 INFO [Listener at localhost/46151] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/cluster_90ed3790-4073-b197-f2f1-bf5abc664ba4, deleteOnExit=true 2023-07-12 15:04:26,432 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-12 15:04:26,432 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/test.cache.data in system properties and HBase conf 2023-07-12 15:04:26,432 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 15:04:26,432 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/hadoop.log.dir in system properties and HBase conf 2023-07-12 15:04:26,432 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 15:04:26,432 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 15:04:26,432 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 15:04:26,433 DEBUG [Listener at localhost/46151] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 15:04:26,433 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 15:04:26,433 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 15:04:26,433 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 15:04:26,433 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 15:04:26,433 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 15:04:26,433 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 15:04:26,433 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 15:04:26,434 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 15:04:26,434 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 15:04:26,434 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/nfs.dump.dir in system properties and HBase conf 2023-07-12 15:04:26,434 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/java.io.tmpdir in system properties and HBase conf 2023-07-12 15:04:26,434 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 15:04:26,434 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 15:04:26,434 INFO [Listener at localhost/46151] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 15:04:26,442 WARN [Listener at localhost/46151] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 15:04:26,443 WARN [Listener at localhost/46151] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 15:04:26,492 WARN [Listener at localhost/46151] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 15:04:26,495 INFO [Listener at localhost/46151] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 15:04:26,544 INFO [Listener at localhost/46151] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/java.io.tmpdir/Jetty_localhost_41981_hdfs____7gfgn3/webapp 2023-07-12 15:04:26,637 INFO [Listener at localhost/46151] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41981 2023-07-12 15:04:26,641 WARN [Listener at localhost/46151] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 15:04:26,642 WARN [Listener at localhost/46151] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 15:04:26,680 WARN [Listener at localhost/35443] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 15:04:26,691 WARN [Listener at localhost/35443] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-12 15:04:26,721 WARN [Listener at localhost/35443] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 15:04:26,723 WARN [Listener at localhost/35443] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 15:04:26,725 INFO [Listener at localhost/35443] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 15:04:26,730 INFO [Listener at localhost/35443] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/java.io.tmpdir/Jetty_localhost_36501_datanode____.b4gnae/webapp 2023-07-12 15:04:26,823 INFO [Listener at localhost/35443] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36501 2023-07-12 15:04:26,829 WARN [Listener at localhost/38029] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 15:04:26,927 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8d9b897374d00acd: Processing first storage report for DS-c47add32-855e-45e9-892b-b126e5589b10 from datanode 791d1ad3-98cc-4198-a234-45ded6a5adc6 2023-07-12 15:04:26,927 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8d9b897374d00acd: from storage DS-c47add32-855e-45e9-892b-b126e5589b10 node DatanodeRegistration(127.0.0.1:44635, datanodeUuid=791d1ad3-98cc-4198-a234-45ded6a5adc6, infoPort=37119, infoSecurePort=0, ipcPort=38029, storageInfo=lv=-57;cid=testClusterID;nsid=359672691;c=1689174266449), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-12 15:04:26,927 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8d9b897374d00acd: Processing first storage report for DS-d7c7b7d9-ecff-4669-849b-99cd4ec525fb from datanode 791d1ad3-98cc-4198-a234-45ded6a5adc6 2023-07-12 15:04:26,927 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8d9b897374d00acd: from storage DS-d7c7b7d9-ecff-4669-849b-99cd4ec525fb node DatanodeRegistration(127.0.0.1:44635, datanodeUuid=791d1ad3-98cc-4198-a234-45ded6a5adc6, infoPort=37119, infoSecurePort=0, ipcPort=38029, storageInfo=lv=-57;cid=testClusterID;nsid=359672691;c=1689174266449), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 15:04:26,936 DEBUG [Listener at localhost/38029] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35 2023-07-12 15:04:26,937 DEBUG [Listener at localhost/38029] zookeeper.MiniZooKeeperCluster(243): Failed binding ZK Server to client port: 53130 java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:461) at sun.nio.ch.Net.bind(Net.java:453) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:78) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:687) at org.apache.zookeeper.server.ServerCnxnFactory.configure(ServerCnxnFactory.java:76) at org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:239) at org.apache.hadoop.hbase.HBaseZKTestingUtility.startMiniZKCluster(HBaseZKTestingUtility.java:129) at org.apache.hadoop.hbase.HBaseZKTestingUtility.startMiniZKCluster(HBaseZKTestingUtility.java:102) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1090) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.setup(TestMasterFailoverWithProcedures.java:78) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:04:26,939 INFO [Listener at localhost/38029] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/cluster_90ed3790-4073-b197-f2f1-bf5abc664ba4/zookeeper_0, clientPort=53131, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/cluster_90ed3790-4073-b197-f2f1-bf5abc664ba4/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/cluster_90ed3790-4073-b197-f2f1-bf5abc664ba4/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 15:04:26,941 INFO [Listener at localhost/38029] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=53131 2023-07-12 15:04:26,941 INFO [Listener at localhost/38029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:26,942 INFO [Listener at localhost/38029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:27,353 INFO [Listener at localhost/38029] util.FSUtils(471): Created version file at hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb with version=8 2023-07-12 15:04:27,353 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/hbase-staging 2023-07-12 15:04:27,355 INFO [Listener at localhost/38029] client.ConnectionUtils(127): master/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:04:27,355 INFO [Listener at localhost/38029] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:27,355 INFO [Listener at localhost/38029] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:27,356 INFO [Listener at localhost/38029] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:04:27,356 INFO [Listener at localhost/38029] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:27,356 INFO [Listener at localhost/38029] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:04:27,356 INFO [Listener at localhost/38029] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:04:27,357 INFO [Listener at localhost/38029] ipc.NettyRpcServer(120): Bind to /172.31.10.131:35575 2023-07-12 15:04:27,357 INFO [Listener at localhost/38029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:27,358 INFO [Listener at localhost/38029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:27,360 INFO [Listener at localhost/38029] zookeeper.RecoverableZooKeeper(93): Process identifier=master:35575 connecting to ZooKeeper ensemble=127.0.0.1:53131 2023-07-12 15:04:27,372 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:355750x0, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:04:27,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:35575-0x101c5ac9e960000 connected 2023-07-12 15:04:27,389 DEBUG [Listener at localhost/38029] zookeeper.ZKUtil(164): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:27,389 DEBUG [Listener at localhost/38029] zookeeper.ZKUtil(164): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:27,390 DEBUG [Listener at localhost/38029] zookeeper.ZKUtil(164): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:04:27,390 DEBUG [Listener at localhost/38029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35575 2023-07-12 15:04:27,390 DEBUG [Listener at localhost/38029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35575 2023-07-12 15:04:27,390 DEBUG [Listener at localhost/38029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35575 2023-07-12 15:04:27,391 DEBUG [Listener at localhost/38029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35575 2023-07-12 15:04:27,391 DEBUG [Listener at localhost/38029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35575 2023-07-12 15:04:27,391 INFO [Listener at localhost/38029] master.HMaster(444): hbase.rootdir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb, hbase.cluster.distributed=false 2023-07-12 15:04:27,392 INFO [Listener at localhost/38029] client.ConnectionUtils(127): master/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:04:27,393 INFO [Listener at localhost/38029] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:27,393 INFO [Listener at localhost/38029] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:27,393 INFO [Listener at localhost/38029] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:04:27,393 INFO [Listener at localhost/38029] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:27,393 INFO [Listener at localhost/38029] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:04:27,393 INFO [Listener at localhost/38029] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:04:27,394 INFO [Listener at localhost/38029] ipc.NettyRpcServer(120): Bind to /172.31.10.131:39873 2023-07-12 15:04:27,395 INFO [Listener at localhost/38029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:27,396 INFO [Listener at localhost/38029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:27,396 INFO [Listener at localhost/38029] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39873 connecting to ZooKeeper ensemble=127.0.0.1:53131 2023-07-12 15:04:27,399 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:398730x0, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:04:27,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:39873-0x101c5ac9e960001 connected 2023-07-12 15:04:27,402 DEBUG [Listener at localhost/38029] zookeeper.ZKUtil(164): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:27,403 DEBUG [Listener at localhost/38029] zookeeper.ZKUtil(164): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:27,403 DEBUG [Listener at localhost/38029] zookeeper.ZKUtil(164): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:04:27,404 DEBUG [Listener at localhost/38029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39873 2023-07-12 15:04:27,404 DEBUG [Listener at localhost/38029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39873 2023-07-12 15:04:27,404 DEBUG [Listener at localhost/38029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39873 2023-07-12 15:04:27,411 DEBUG [Listener at localhost/38029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39873 2023-07-12 15:04:27,411 DEBUG [Listener at localhost/38029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39873 2023-07-12 15:04:27,411 INFO [Listener at localhost/38029] master.HMaster(444): hbase.rootdir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb, hbase.cluster.distributed=false 2023-07-12 15:04:27,424 INFO [Listener at localhost/38029] client.ConnectionUtils(127): regionserver/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:04:27,424 INFO [Listener at localhost/38029] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:27,424 INFO [Listener at localhost/38029] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:27,424 INFO [Listener at localhost/38029] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:04:27,424 INFO [Listener at localhost/38029] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:27,424 INFO [Listener at localhost/38029] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:04:27,424 INFO [Listener at localhost/38029] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:04:27,425 INFO [Listener at localhost/38029] ipc.NettyRpcServer(120): Bind to /172.31.10.131:44477 2023-07-12 15:04:27,426 INFO [Listener at localhost/38029] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 15:04:27,426 DEBUG [Listener at localhost/38029] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 15:04:27,427 INFO [Listener at localhost/38029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:27,428 INFO [Listener at localhost/38029] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:27,429 INFO [Listener at localhost/38029] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:44477 connecting to ZooKeeper ensemble=127.0.0.1:53131 2023-07-12 15:04:27,431 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): regionserver:444770x0, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:04:27,432 DEBUG [Listener at localhost/38029] zookeeper.ZKUtil(164): regionserver:444770x0, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:27,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:44477-0x101c5ac9e960002 connected 2023-07-12 15:04:27,432 DEBUG [Listener at localhost/38029] zookeeper.ZKUtil(164): regionserver:44477-0x101c5ac9e960002, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:27,433 DEBUG [Listener at localhost/38029] zookeeper.ZKUtil(164): regionserver:44477-0x101c5ac9e960002, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:04:27,433 DEBUG [Listener at localhost/38029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44477 2023-07-12 15:04:27,433 DEBUG [Listener at localhost/38029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44477 2023-07-12 15:04:27,434 DEBUG [Listener at localhost/38029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44477 2023-07-12 15:04:27,434 DEBUG [Listener at localhost/38029] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44477 2023-07-12 15:04:27,434 DEBUG [Listener at localhost/38029] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44477 2023-07-12 15:04:27,435 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase5.apache.org,35575,1689174267355 2023-07-12 15:04:27,436 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase5.apache.org,39873,1689174267392 2023-07-12 15:04:27,438 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:27,438 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:27,439 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase5.apache.org,35575,1689174267355 2023-07-12 15:04:27,439 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase5.apache.org,39873,1689174267392 2023-07-12 15:04:27,440 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:27,440 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:27,440 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): regionserver:44477-0x101c5ac9e960002, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:27,440 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:27,440 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:27,441 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:27,442 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:27,442 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase5.apache.org,35575,1689174267355 from backup master directory 2023-07-12 15:04:27,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:27,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:27,443 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase5.apache.org,35575,1689174267355; waiting to become the next active master 2023-07-12 15:04:27,445 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase5.apache.org,35575,1689174267355 2023-07-12 15:04:27,445 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:27,445 WARN [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:04:27,445 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase5.apache.org,35575,1689174267355 2023-07-12 15:04:27,445 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:27,858 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/hbase.id with ID: 7251e262-1761-41ff-a1b0-98701e570a40 2023-07-12 15:04:27,870 INFO [master/jenkins-hbase5:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:27,873 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:27,873 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:28,284 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x51025d1d to 127.0.0.1:53131 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:28,287 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28c4d819, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:28,288 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:28,288 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 15:04:28,288 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:28,290 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store-tmp 2023-07-12 15:04:28,696 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:28,697 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:04:28,697 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:28,697 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:28,697 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:04:28,697 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:28,697 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:28,697 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:28,698 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/WALs/jenkins-hbase5.apache.org,35575,1689174267355 2023-07-12 15:04:28,700 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C35575%2C1689174267355, suffix=, logDir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/WALs/jenkins-hbase5.apache.org,35575,1689174267355, archiveDir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/oldWALs, maxLogs=10 2023-07-12 15:04:28,714 DEBUG [RS-EventLoopGroup-11-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44635,DS-c47add32-855e-45e9-892b-b126e5589b10,DISK] 2023-07-12 15:04:28,717 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/WALs/jenkins-hbase5.apache.org,35575,1689174267355/jenkins-hbase5.apache.org%2C35575%2C1689174267355.1689174268701 2023-07-12 15:04:28,717 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44635,DS-c47add32-855e-45e9-892b-b126e5589b10,DISK]] 2023-07-12 15:04:28,717 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:28,717 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:28,718 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:28,718 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:28,719 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:28,720 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 15:04:28,721 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 15:04:28,721 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:28,722 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:28,722 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:28,725 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:28,727 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:28,728 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61778907, jitterRate=-0.07942254841327667}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:04:28,728 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:28,728 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 15:04:28,729 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:28,729 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:28,729 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:28,730 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:28,730 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-07-12 15:04:28,730 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 15:04:28,731 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 15:04:28,732 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 15:04:28,742 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 15:04:28,742 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 15:04:28,743 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 15:04:28,743 INFO [master/jenkins-hbase5:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 15:04:28,743 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 15:04:28,745 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:28,745 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:28,746 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 15:04:28,746 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 15:04:28,747 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 15:04:28,749 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): regionserver:44477-0x101c5ac9e960002, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:28,749 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:28,749 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:28,749 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:28,749 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:28,750 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase5.apache.org,35575,1689174267355, sessionid=0x101c5ac9e960000, setting cluster-up flag (Was=false) 2023-07-12 15:04:28,757 INFO [M:1;jenkins-hbase5:39873] regionserver.HRegionServer(951): ClusterId : 7251e262-1761-41ff-a1b0-98701e570a40 2023-07-12 15:04:28,758 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:28,758 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:28,764 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 15:04:28,764 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,35575,1689174267355 2023-07-12 15:04:28,767 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:28,767 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:28,772 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 15:04:28,772 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,35575,1689174267355 2023-07-12 15:04:28,773 WARN [master/jenkins-hbase5:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.hbase-snapshot/.tmp 2023-07-12 15:04:28,777 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:28,777 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:28,777 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:28,777 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:28,777 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:28,777 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase5:0, corePoolSize=10, maxPoolSize=10 2023-07-12 15:04:28,777 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:28,777 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:04:28,777 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:28,777 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174298777 2023-07-12 15:04:28,778 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 15:04:28,778 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 15:04:28,778 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 15:04:28,778 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 15:04:28,778 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 15:04:28,778 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 15:04:28,779 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:28,779 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 15:04:28,779 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 15:04:28,779 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 15:04:28,779 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:28,780 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 15:04:28,780 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 15:04:28,780 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 15:04:28,780 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174268780,5,FailOnTimeoutGroup] 2023-07-12 15:04:28,780 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174268780,5,FailOnTimeoutGroup] 2023-07-12 15:04:28,780 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:28,780 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 15:04:28,780 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:28,780 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:28,781 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:28,837 INFO [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(951): ClusterId : 7251e262-1761-41ff-a1b0-98701e570a40 2023-07-12 15:04:28,837 DEBUG [RS:0;jenkins-hbase5:44477] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 15:04:28,839 DEBUG [RS:0;jenkins-hbase5:44477] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 15:04:28,839 DEBUG [RS:0;jenkins-hbase5:44477] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 15:04:28,842 DEBUG [RS:0;jenkins-hbase5:44477] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 15:04:28,843 DEBUG [RS:0;jenkins-hbase5:44477] zookeeper.ReadOnlyZKClient(139): Connect 0x261edccd to 127.0.0.1:53131 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:28,847 DEBUG [RS:0;jenkins-hbase5:44477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ce6b7bd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:28,847 DEBUG [RS:0;jenkins-hbase5:44477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69e42c45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:04:28,856 DEBUG [RS:0;jenkins-hbase5:44477] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase5:44477 2023-07-12 15:04:28,856 INFO [RS:0;jenkins-hbase5:44477] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 15:04:28,856 INFO [RS:0;jenkins-hbase5:44477] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 15:04:28,856 DEBUG [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 15:04:28,856 INFO [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase5.apache.org,35575,1689174267355 with isa=jenkins-hbase5.apache.org/172.31.10.131:44477, startcode=1689174267423 2023-07-12 15:04:28,856 DEBUG [RS:0;jenkins-hbase5:44477] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 15:04:28,859 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:45123, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 15:04:28,860 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=35575] master.ServerManager(394): Registering regionserver=jenkins-hbase5.apache.org,44477,1689174267423 2023-07-12 15:04:28,861 DEBUG [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb 2023-07-12 15:04:28,861 DEBUG [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:35443 2023-07-12 15:04:28,861 DEBUG [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 15:04:28,862 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:04:28,862 DEBUG [RS:0;jenkins-hbase5:44477] zookeeper.ZKUtil(162): regionserver:44477-0x101c5ac9e960002, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,44477,1689174267423 2023-07-12 15:04:28,862 WARN [RS:0;jenkins-hbase5:44477] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:04:28,863 INFO [RS:0;jenkins-hbase5:44477] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:28,863 DEBUG [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1948): logDir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/WALs/jenkins-hbase5.apache.org,44477,1689174267423 2023-07-12 15:04:28,863 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase5.apache.org,44477,1689174267423] 2023-07-12 15:04:28,867 DEBUG [RS:0;jenkins-hbase5:44477] zookeeper.ZKUtil(162): regionserver:44477-0x101c5ac9e960002, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,44477,1689174267423 2023-07-12 15:04:28,868 DEBUG [RS:0;jenkins-hbase5:44477] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 15:04:28,868 INFO [RS:0;jenkins-hbase5:44477] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 15:04:28,869 INFO [RS:0;jenkins-hbase5:44477] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 15:04:28,870 INFO [RS:0;jenkins-hbase5:44477] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:04:28,870 INFO [RS:0;jenkins-hbase5:44477] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:28,871 INFO [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 15:04:28,872 INFO [RS:0;jenkins-hbase5:44477] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:28,872 DEBUG [RS:0;jenkins-hbase5:44477] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:28,872 DEBUG [RS:0;jenkins-hbase5:44477] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:28,872 DEBUG [RS:0;jenkins-hbase5:44477] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:28,872 DEBUG [RS:0;jenkins-hbase5:44477] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:28,872 DEBUG [RS:0;jenkins-hbase5:44477] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:28,872 DEBUG [RS:0;jenkins-hbase5:44477] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:04:28,872 DEBUG [RS:0;jenkins-hbase5:44477] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:28,872 DEBUG [RS:0;jenkins-hbase5:44477] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:28,873 DEBUG [RS:0;jenkins-hbase5:44477] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:28,873 DEBUG [RS:0;jenkins-hbase5:44477] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:28,876 INFO [RS:0;jenkins-hbase5:44477] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:28,876 INFO [RS:0;jenkins-hbase5:44477] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:28,876 INFO [RS:0;jenkins-hbase5:44477] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:28,890 INFO [RS:0;jenkins-hbase5:44477] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 15:04:28,890 INFO [RS:0;jenkins-hbase5:44477] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,44477,1689174267423-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:28,900 INFO [RS:0;jenkins-hbase5:44477] regionserver.Replication(203): jenkins-hbase5.apache.org,44477,1689174267423 started 2023-07-12 15:04:28,900 INFO [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1637): Serving as jenkins-hbase5.apache.org,44477,1689174267423, RpcServer on jenkins-hbase5.apache.org/172.31.10.131:44477, sessionid=0x101c5ac9e960002 2023-07-12 15:04:28,901 DEBUG [RS:0;jenkins-hbase5:44477] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 15:04:28,901 DEBUG [RS:0;jenkins-hbase5:44477] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase5.apache.org,44477,1689174267423 2023-07-12 15:04:28,901 DEBUG [RS:0;jenkins-hbase5:44477] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,44477,1689174267423' 2023-07-12 15:04:28,901 DEBUG [RS:0;jenkins-hbase5:44477] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:04:28,901 DEBUG [RS:0;jenkins-hbase5:44477] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:04:28,901 DEBUG [RS:0;jenkins-hbase5:44477] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 15:04:28,901 DEBUG [RS:0;jenkins-hbase5:44477] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 15:04:28,901 DEBUG [RS:0;jenkins-hbase5:44477] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase5.apache.org,44477,1689174267423 2023-07-12 15:04:28,901 DEBUG [RS:0;jenkins-hbase5:44477] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,44477,1689174267423' 2023-07-12 15:04:28,901 DEBUG [RS:0;jenkins-hbase5:44477] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 15:04:28,902 DEBUG [RS:0;jenkins-hbase5:44477] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 15:04:28,902 DEBUG [RS:0;jenkins-hbase5:44477] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 15:04:28,902 INFO [RS:0;jenkins-hbase5:44477] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 15:04:28,902 INFO [RS:0;jenkins-hbase5:44477] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 15:04:29,004 INFO [RS:0;jenkins-hbase5:44477] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C44477%2C1689174267423, suffix=, logDir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/WALs/jenkins-hbase5.apache.org,44477,1689174267423, archiveDir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/oldWALs, maxLogs=32 2023-07-12 15:04:29,021 DEBUG [RS-EventLoopGroup-11-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44635,DS-c47add32-855e-45e9-892b-b126e5589b10,DISK] 2023-07-12 15:04:29,023 INFO [RS:0;jenkins-hbase5:44477] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/WALs/jenkins-hbase5.apache.org,44477,1689174267423/jenkins-hbase5.apache.org%2C44477%2C1689174267423.1689174269005 2023-07-12 15:04:29,023 DEBUG [RS:0;jenkins-hbase5:44477] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44635,DS-c47add32-855e-45e9-892b-b126e5589b10,DISK]] 2023-07-12 15:04:29,189 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:29,190 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:29,190 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb 2023-07-12 15:04:29,597 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:29,598 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 15:04:29,600 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/info 2023-07-12 15:04:29,600 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 15:04:29,601 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:29,601 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 15:04:29,602 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:04:29,603 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 15:04:29,603 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:29,604 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 15:04:29,605 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/table 2023-07-12 15:04:29,605 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 15:04:29,606 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:29,606 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740 2023-07-12 15:04:29,607 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740 2023-07-12 15:04:29,609 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 15:04:29,611 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 15:04:29,613 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:29,613 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61466058, jitterRate=-0.08408436179161072}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 15:04:29,613 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 15:04:29,613 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 15:04:29,613 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 15:04:29,613 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 15:04:29,613 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 15:04:29,613 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 15:04:29,614 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 15:04:29,614 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 15:04:29,615 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:29,615 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 15:04:29,615 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 15:04:29,617 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 15:04:29,618 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 15:04:29,768 DEBUG [jenkins-hbase5:35575] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 15:04:29,769 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,44477,1689174267423, state=OPENING 2023-07-12 15:04:29,770 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 15:04:29,772 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:29,772 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:29,772 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:04:29,772 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:04:29,772 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,44477,1689174267423}] 2023-07-12 15:04:29,925 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase5.apache.org,44477,1689174267423 2023-07-12 15:04:29,926 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 15:04:29,929 INFO [RS-EventLoopGroup-11-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37364, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 15:04:29,933 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 15:04:29,933 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:29,935 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C44477%2C1689174267423.meta, suffix=.meta, logDir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/WALs/jenkins-hbase5.apache.org,44477,1689174267423, archiveDir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/oldWALs, maxLogs=32 2023-07-12 15:04:29,951 DEBUG [RS-EventLoopGroup-11-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44635,DS-c47add32-855e-45e9-892b-b126e5589b10,DISK] 2023-07-12 15:04:29,953 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/WALs/jenkins-hbase5.apache.org,44477,1689174267423/jenkins-hbase5.apache.org%2C44477%2C1689174267423.meta.1689174269936.meta 2023-07-12 15:04:29,953 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44635,DS-c47add32-855e-45e9-892b-b126e5589b10,DISK]] 2023-07-12 15:04:29,953 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:29,953 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 15:04:29,953 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 15:04:29,954 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 15:04:29,954 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 15:04:29,954 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:29,954 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 15:04:29,954 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 15:04:29,955 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 15:04:29,956 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/info 2023-07-12 15:04:29,956 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/info 2023-07-12 15:04:29,956 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 15:04:29,957 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:29,957 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 15:04:29,958 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:04:29,958 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:04:29,958 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 15:04:29,959 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:29,959 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 15:04:29,960 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/table 2023-07-12 15:04:29,960 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/table 2023-07-12 15:04:29,960 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 15:04:29,961 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:29,962 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740 2023-07-12 15:04:29,963 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740 2023-07-12 15:04:29,965 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 15:04:29,966 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 15:04:29,967 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63771691, jitterRate=-0.049727752804756165}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 15:04:29,967 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 15:04:29,968 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689174269925 2023-07-12 15:04:29,971 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 15:04:29,971 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 15:04:29,972 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,44477,1689174267423, state=OPEN 2023-07-12 15:04:29,977 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 15:04:29,977 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 15:04:29,977 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:04:29,977 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:04:29,979 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 15:04:29,979 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,44477,1689174267423 in 205 msec 2023-07-12 15:04:29,981 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 15:04:29,982 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 364 msec 2023-07-12 15:04:29,984 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.2060 sec 2023-07-12 15:04:29,984 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689174269984, completionTime=-1 2023-07-12 15:04:29,984 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 15:04:29,984 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:29,986 DEBUG [hconnection-0x103b85f7-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:04:29,988 INFO [RS-EventLoopGroup-11-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37376, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:04:29,990 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:29,990 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174329990 2023-07-12 15:04:29,990 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174389990 2023-07-12 15:04:29,990 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-07-12 15:04:29,997 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,35575,1689174267355-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:29,997 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,35575,1689174267355-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:29,997 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,35575,1689174267355-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:29,998 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase5:35575, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:29,998 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:29,998 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 15:04:29,998 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:29,999 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:30,000 DEBUG [master/jenkins-hbase5:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 15:04:30,001 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:04:30,002 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:04:30,004 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/hbase/namespace/d5915cc2af7dcf8b3ba85977de3a2d50 2023-07-12 15:04:30,005 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/hbase/namespace/d5915cc2af7dcf8b3ba85977de3a2d50 empty. 2023-07-12 15:04:30,005 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/hbase/namespace/d5915cc2af7dcf8b3ba85977de3a2d50 2023-07-12 15:04:30,005 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 15:04:30,024 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:30,025 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => d5915cc2af7dcf8b3ba85977de3a2d50, NAME => 'hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp 2023-07-12 15:04:30,435 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:30,435 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing d5915cc2af7dcf8b3ba85977de3a2d50, disabling compactions & flushes 2023-07-12 15:04:30,435 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50. 2023-07-12 15:04:30,435 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50. 2023-07-12 15:04:30,435 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50. after waiting 0 ms 2023-07-12 15:04:30,435 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50. 2023-07-12 15:04:30,435 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50. 2023-07-12 15:04:30,435 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for d5915cc2af7dcf8b3ba85977de3a2d50: 2023-07-12 15:04:30,438 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:04:30,439 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689174270439"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174270439"}]},"ts":"1689174270439"} 2023-07-12 15:04:30,441 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 15:04:30,442 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:04:30,442 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174270442"}]},"ts":"1689174270442"} 2023-07-12 15:04:30,443 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 15:04:30,448 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d5915cc2af7dcf8b3ba85977de3a2d50, ASSIGN}] 2023-07-12 15:04:30,450 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d5915cc2af7dcf8b3ba85977de3a2d50, ASSIGN 2023-07-12 15:04:30,451 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=d5915cc2af7dcf8b3ba85977de3a2d50, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,44477,1689174267423; forceNewPlan=false, retain=false 2023-07-12 15:04:30,602 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=d5915cc2af7dcf8b3ba85977de3a2d50, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,44477,1689174267423 2023-07-12 15:04:30,602 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689174270602"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174270602"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174270602"}]},"ts":"1689174270602"} 2023-07-12 15:04:30,604 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure d5915cc2af7dcf8b3ba85977de3a2d50, server=jenkins-hbase5.apache.org,44477,1689174267423}] 2023-07-12 15:04:30,760 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50. 2023-07-12 15:04:30,760 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d5915cc2af7dcf8b3ba85977de3a2d50, NAME => 'hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:30,760 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace d5915cc2af7dcf8b3ba85977de3a2d50 2023-07-12 15:04:30,761 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:30,761 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for d5915cc2af7dcf8b3ba85977de3a2d50 2023-07-12 15:04:30,761 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for d5915cc2af7dcf8b3ba85977de3a2d50 2023-07-12 15:04:30,762 INFO [StoreOpener-d5915cc2af7dcf8b3ba85977de3a2d50-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region d5915cc2af7dcf8b3ba85977de3a2d50 2023-07-12 15:04:30,763 DEBUG [StoreOpener-d5915cc2af7dcf8b3ba85977de3a2d50-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/namespace/d5915cc2af7dcf8b3ba85977de3a2d50/info 2023-07-12 15:04:30,763 DEBUG [StoreOpener-d5915cc2af7dcf8b3ba85977de3a2d50-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/namespace/d5915cc2af7dcf8b3ba85977de3a2d50/info 2023-07-12 15:04:30,764 INFO [StoreOpener-d5915cc2af7dcf8b3ba85977de3a2d50-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d5915cc2af7dcf8b3ba85977de3a2d50 columnFamilyName info 2023-07-12 15:04:30,764 INFO [StoreOpener-d5915cc2af7dcf8b3ba85977de3a2d50-1] regionserver.HStore(310): Store=d5915cc2af7dcf8b3ba85977de3a2d50/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:30,765 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/namespace/d5915cc2af7dcf8b3ba85977de3a2d50 2023-07-12 15:04:30,765 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/namespace/d5915cc2af7dcf8b3ba85977de3a2d50 2023-07-12 15:04:30,768 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for d5915cc2af7dcf8b3ba85977de3a2d50 2023-07-12 15:04:30,771 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/namespace/d5915cc2af7dcf8b3ba85977de3a2d50/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:30,772 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened d5915cc2af7dcf8b3ba85977de3a2d50; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66855582, jitterRate=-0.003774195909500122}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:04:30,772 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for d5915cc2af7dcf8b3ba85977de3a2d50: 2023-07-12 15:04:30,773 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50., pid=6, masterSystemTime=1689174270756 2023-07-12 15:04:30,775 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50. 2023-07-12 15:04:30,775 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50. 2023-07-12 15:04:30,775 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=d5915cc2af7dcf8b3ba85977de3a2d50, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,44477,1689174267423 2023-07-12 15:04:30,775 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689174270775"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174270775"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174270775"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174270775"}]},"ts":"1689174270775"} 2023-07-12 15:04:30,779 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 15:04:30,779 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure d5915cc2af7dcf8b3ba85977de3a2d50, server=jenkins-hbase5.apache.org,44477,1689174267423 in 173 msec 2023-07-12 15:04:30,781 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 15:04:30,781 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=d5915cc2af7dcf8b3ba85977de3a2d50, ASSIGN in 332 msec 2023-07-12 15:04:30,782 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 15:04:30,782 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174270782"}]},"ts":"1689174270782"} 2023-07-12 15:04:30,783 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 15:04:30,786 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 15:04:30,787 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 788 msec 2023-07-12 15:04:30,801 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 15:04:30,802 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:04:30,802 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:30,802 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:30,806 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:30,817 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:04:30,820 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 14 msec 2023-07-12 15:04:30,828 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:30,836 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:04:30,839 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 11 msec 2023-07-12 15:04:30,853 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 15:04:30,855 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 15:04:30,855 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.410sec 2023-07-12 15:04:30,856 INFO [master/jenkins-hbase5:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 15:04:30,856 INFO [master/jenkins-hbase5:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 15:04:30,856 INFO [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 15:04:30,856 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,35575,1689174267355-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 15:04:30,856 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,35575,1689174267355-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 15:04:30,857 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 15:04:30,938 DEBUG [Listener at localhost/38029] zookeeper.ReadOnlyZKClient(139): Connect 0x71f57cde to 127.0.0.1:53131 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:30,946 DEBUG [Listener at localhost/38029] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49e4faaa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:30,948 DEBUG [hconnection-0x3507d923-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:04:30,949 INFO [RS-EventLoopGroup-11-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37388, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:04:30,951 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase5.apache.org,35575,1689174267355 2023-07-12 15:04:30,951 WARN [Listener at localhost/38029] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-12 15:04:30,954 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:30,956 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:04:30,956 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:04:30,958 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4 2023-07-12 15:04:30,959 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4 empty. 2023-07-12 15:04:30,960 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4 2023-07-12 15:04:30,960 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testDeleteWithFailoverAtStep4 regions 2023-07-12 15:04:31,370 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/default/testDeleteWithFailoverAtStep4/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:31,372 INFO [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(7675): creating {ENCODED => a2058f5c250e868b28bffe4ac4da36a4, NAME => 'testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testDeleteWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp 2023-07-12 15:04:31,781 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(866): Instantiated testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:31,781 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1604): Closing a2058f5c250e868b28bffe4ac4da36a4, disabling compactions & flushes 2023-07-12 15:04:31,781 INFO [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1626): Closing region testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4. 2023-07-12 15:04:31,781 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4. 2023-07-12 15:04:31,781 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1714): Acquired close lock on testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4. after waiting 0 ms 2023-07-12 15:04:31,781 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1724): Updates disabled for region testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4. 2023-07-12 15:04:31,781 INFO [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1838): Closed testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4. 2023-07-12 15:04:31,781 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1558): Region close journal for a2058f5c250e868b28bffe4ac4da36a4: 2023-07-12 15:04:31,784 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:04:31,784 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689174271784"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174271784"}]},"ts":"1689174271784"} 2023-07-12 15:04:31,786 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 15:04:31,787 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:04:31,787 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174271787"}]},"ts":"1689174271787"} 2023-07-12 15:04:31,789 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteWithFailoverAtStep4, state=ENABLING in hbase:meta 2023-07-12 15:04:31,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=a2058f5c250e868b28bffe4ac4da36a4, ASSIGN}] 2023-07-12 15:04:31,793 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=a2058f5c250e868b28bffe4ac4da36a4, ASSIGN 2023-07-12 15:04:31,794 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=a2058f5c250e868b28bffe4ac4da36a4, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,44477,1689174267423; forceNewPlan=false, retain=false 2023-07-12 15:04:31,945 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=a2058f5c250e868b28bffe4ac4da36a4, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,44477,1689174267423 2023-07-12 15:04:31,945 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689174271945"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174271945"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174271945"}]},"ts":"1689174271945"} 2023-07-12 15:04:31,948 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure a2058f5c250e868b28bffe4ac4da36a4, server=jenkins-hbase5.apache.org,44477,1689174267423}] 2023-07-12 15:04:32,104 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4. 2023-07-12 15:04:32,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a2058f5c250e868b28bffe4ac4da36a4, NAME => 'testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:32,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDeleteWithFailoverAtStep4 a2058f5c250e868b28bffe4ac4da36a4 2023-07-12 15:04:32,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:32,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for a2058f5c250e868b28bffe4ac4da36a4 2023-07-12 15:04:32,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for a2058f5c250e868b28bffe4ac4da36a4 2023-07-12 15:04:32,106 INFO [StoreOpener-a2058f5c250e868b28bffe4ac4da36a4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region a2058f5c250e868b28bffe4ac4da36a4 2023-07-12 15:04:32,107 DEBUG [StoreOpener-a2058f5c250e868b28bffe4ac4da36a4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4/f1 2023-07-12 15:04:32,107 DEBUG [StoreOpener-a2058f5c250e868b28bffe4ac4da36a4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4/f1 2023-07-12 15:04:32,108 INFO [StoreOpener-a2058f5c250e868b28bffe4ac4da36a4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a2058f5c250e868b28bffe4ac4da36a4 columnFamilyName f1 2023-07-12 15:04:32,108 INFO [StoreOpener-a2058f5c250e868b28bffe4ac4da36a4-1] regionserver.HStore(310): Store=a2058f5c250e868b28bffe4ac4da36a4/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:32,108 INFO [StoreOpener-a2058f5c250e868b28bffe4ac4da36a4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region a2058f5c250e868b28bffe4ac4da36a4 2023-07-12 15:04:32,110 DEBUG [StoreOpener-a2058f5c250e868b28bffe4ac4da36a4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4/f2 2023-07-12 15:04:32,110 DEBUG [StoreOpener-a2058f5c250e868b28bffe4ac4da36a4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4/f2 2023-07-12 15:04:32,110 INFO [StoreOpener-a2058f5c250e868b28bffe4ac4da36a4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a2058f5c250e868b28bffe4ac4da36a4 columnFamilyName f2 2023-07-12 15:04:32,110 INFO [StoreOpener-a2058f5c250e868b28bffe4ac4da36a4-1] regionserver.HStore(310): Store=a2058f5c250e868b28bffe4ac4da36a4/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:32,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4 2023-07-12 15:04:32,112 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4 2023-07-12 15:04:32,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testDeleteWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:04:32,115 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for a2058f5c250e868b28bffe4ac4da36a4 2023-07-12 15:04:32,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:32,117 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened a2058f5c250e868b28bffe4ac4da36a4; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63409754, jitterRate=-0.055121034383773804}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:04:32,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for a2058f5c250e868b28bffe4ac4da36a4: 2023-07-12 15:04:32,118 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4., pid=11, masterSystemTime=1689174272100 2023-07-12 15:04:32,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4. 2023-07-12 15:04:32,120 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4. 2023-07-12 15:04:32,120 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=a2058f5c250e868b28bffe4ac4da36a4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,44477,1689174267423 2023-07-12 15:04:32,121 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689174272120"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174272120"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174272120"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174272120"}]},"ts":"1689174272120"} 2023-07-12 15:04:32,124 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-12 15:04:32,124 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure a2058f5c250e868b28bffe4ac4da36a4, server=jenkins-hbase5.apache.org,44477,1689174267423 in 174 msec 2023-07-12 15:04:32,126 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-12 15:04:32,126 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=a2058f5c250e868b28bffe4ac4da36a4, ASSIGN in 332 msec 2023-07-12 15:04:32,126 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 15:04:32,127 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174272126"}]},"ts":"1689174272126"} 2023-07-12 15:04:32,128 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteWithFailoverAtStep4, state=ENABLED in hbase:meta 2023-07-12 15:04:32,130 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 15:04:32,132 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 in 1.1770 sec 2023-07-12 15:04:32,205 DEBUG [Listener at localhost/38029] util.CommonFSUtils(682): File system contents for path hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/default/testDeleteWithFailoverAtStep4 2023-07-12 15:04:32,206 DEBUG [Listener at localhost/38029] util.CommonFSUtils(699): |-.tabledesc/ 2023-07-12 15:04:32,207 DEBUG [Listener at localhost/38029] util.CommonFSUtils(702): |----.tableinfo.0000000001 2023-07-12 15:04:32,207 DEBUG [Listener at localhost/38029] util.CommonFSUtils(699): |-.tmp/ 2023-07-12 15:04:32,207 DEBUG [Listener at localhost/38029] util.CommonFSUtils(699): |-a2058f5c250e868b28bffe4ac4da36a4/ 2023-07-12 15:04:32,208 DEBUG [Listener at localhost/38029] util.CommonFSUtils(702): |----.regioninfo 2023-07-12 15:04:32,208 DEBUG [Listener at localhost/38029] util.CommonFSUtils(699): |----f1/ 2023-07-12 15:04:32,208 DEBUG [Listener at localhost/38029] util.CommonFSUtils(699): |----f2/ 2023-07-12 15:04:32,208 DEBUG [Listener at localhost/38029] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-12 15:04:32,209 DEBUG [Listener at localhost/38029] util.CommonFSUtils(702): |-------1.seqid 2023-07-12 15:04:32,211 DEBUG [Listener at localhost/38029] procedure.MasterProcedureTestingUtility(212): Table directory layout is as expected. 2023-07-12 15:04:32,224 DEBUG [Listener at localhost/38029] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 15:04:32,227 INFO [RS-EventLoopGroup-9-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:46070, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 15:04:32,230 INFO [Listener at localhost/38029] client.HBaseAdmin$15(890): Started disable of testDeleteWithFailoverAtStep4 2023-07-12 15:04:32,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35575] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testDeleteWithFailoverAtStep4 2023-07-12 15:04:32,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35575] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:32,237 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174272237"}]},"ts":"1689174272237"} 2023-07-12 15:04:32,239 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteWithFailoverAtStep4, state=DISABLING in hbase:meta 2023-07-12 15:04:32,242 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testDeleteWithFailoverAtStep4 to state=DISABLING 2023-07-12 15:04:32,243 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=a2058f5c250e868b28bffe4ac4da36a4, UNASSIGN}] 2023-07-12 15:04:32,243 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=a2058f5c250e868b28bffe4ac4da36a4, UNASSIGN 2023-07-12 15:04:32,244 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=a2058f5c250e868b28bffe4ac4da36a4, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,44477,1689174267423 2023-07-12 15:04:32,244 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689174272244"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174272244"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174272244"}]},"ts":"1689174272244"} 2023-07-12 15:04:32,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35575] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 15:04:32,246 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure a2058f5c250e868b28bffe4ac4da36a4, server=jenkins-hbase5.apache.org,44477,1689174267423}] 2023-07-12 15:04:32,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35575] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 15:04:32,398 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close a2058f5c250e868b28bffe4ac4da36a4 2023-07-12 15:04:32,399 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing a2058f5c250e868b28bffe4ac4da36a4, disabling compactions & flushes 2023-07-12 15:04:32,399 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4. 2023-07-12 15:04:32,399 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4. 2023-07-12 15:04:32,399 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4. after waiting 0 ms 2023-07-12 15:04:32,399 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4. 2023-07-12 15:04:32,403 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 15:04:32,404 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4. 2023-07-12 15:04:32,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for a2058f5c250e868b28bffe4ac4da36a4: 2023-07-12 15:04:32,406 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed a2058f5c250e868b28bffe4ac4da36a4 2023-07-12 15:04:32,406 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=a2058f5c250e868b28bffe4ac4da36a4, regionState=CLOSED 2023-07-12 15:04:32,407 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDeleteWithFailoverAtStep4,,1689174270953.a2058f5c250e868b28bffe4ac4da36a4.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689174272406"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174272406"}]},"ts":"1689174272406"} 2023-07-12 15:04:32,410 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-12 15:04:32,410 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure a2058f5c250e868b28bffe4ac4da36a4, server=jenkins-hbase5.apache.org,44477,1689174267423 in 162 msec 2023-07-12 15:04:32,411 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-12 15:04:32,412 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=a2058f5c250e868b28bffe4ac4da36a4, UNASSIGN in 167 msec 2023-07-12 15:04:32,412 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174272412"}]},"ts":"1689174272412"} 2023-07-12 15:04:32,414 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteWithFailoverAtStep4, state=DISABLED in hbase:meta 2023-07-12 15:04:32,415 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testDeleteWithFailoverAtStep4 to state=DISABLED 2023-07-12 15:04:32,417 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 in 182 msec 2023-07-12 15:04:32,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35575] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 15:04:32,551 INFO [Listener at localhost/38029] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testDeleteWithFailoverAtStep4, procId: 12 completed 2023-07-12 15:04:32,551 WARN [Listener at localhost/38029] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-07-12 15:04:32,557 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:32,559 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:32,559 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:32,560 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:32,560 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:32,560 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:32,807 INFO [Listener at localhost/38029] procedure.TestMasterFailoverWithProcedures(306): Restart 0 exec state: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:32,807 INFO [Listener at localhost/38029] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:04:32,807 INFO [Listener at localhost/38029] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:32,807 INFO [Listener at localhost/38029] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:32,809 ERROR [Listener at localhost/38029] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:32,809 INFO [Listener at localhost/38029] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:04:32,809 INFO [Listener at localhost/38029] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:32,809 INFO [Listener at localhost/38029] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:32,809 INFO [Listener at localhost/38029] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:32,810 INFO [Listener at localhost/38029] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:32,811 DEBUG [Listener at localhost/38029] store.ProcedureTree(114): Procedure Procedure(pid=15, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure) stack ids=[] 2023-07-12 15:04:32,811 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:32,812 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:32,812 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:32,812 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:32,812 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:32,812 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:32,813 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(468): Loading pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:32,813 DEBUG [Listener at localhost/38029] procedure2.Procedure(966): pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:32,813 INFO [Listener at localhost/38029] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 15:04:32,813 DEBUG [Listener at localhost/38029] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:32,814 DEBUG [Listener at localhost/38029] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,44477,1689174267423, table=hbase:meta, region=1588230740 2023-07-12 15:04:32,814 INFO [Listener at localhost/38029] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174302814 2023-07-12 15:04:32,815 DEBUG [Listener at localhost/38029] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:32,815 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:32,818 DEBUG [Listener at localhost/38029] assignment.RegionStateStore(147): Load hbase:meta entry region=d5915cc2af7dcf8b3ba85977de3a2d50, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,44477,1689174267423, regionLocation=jenkins-hbase5.apache.org,44477,1689174267423, openSeqNum=2 2023-07-12 15:04:32,818 DEBUG [Listener at localhost/38029] assignment.RegionStateStore(147): Load hbase:meta entry region=a2058f5c250e868b28bffe4ac4da36a4, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,44477,1689174267423, regionLocation=jenkins-hbase5.apache.org,44477,1689174267423, openSeqNum=2 2023-07-12 15:04:32,818 INFO [Listener at localhost/38029] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:32,818 INFO [Listener at localhost/38029] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174332818 2023-07-12 15:04:32,818 INFO [Listener at localhost/38029] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174392818 2023-07-12 15:04:32,818 INFO [Listener at localhost/38029] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 15:04:32,818 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:32,819 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:04:32,820 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:32,824 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4 2023-07-12 15:04:32,826 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4/f1, FileablePath, hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4/f2, FileablePath, hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4/recovered.edits] 2023-07-12 15:04:32,833 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4/recovered.edits/4.seqid to hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/archive/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4/recovered.edits/4.seqid 2023-07-12 15:04:32,834 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4 2023-07-12 15:04:32,834 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testDeleteWithFailoverAtStep4 regions 2023-07-12 15:04:32,835 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:32,835 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:32,835 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:32,835 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:33,068 INFO [Listener at localhost/38029] procedure.TestMasterFailoverWithProcedures(306): Restart 1 exec state: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,069 INFO [Listener at localhost/38029] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:04:33,069 INFO [Listener at localhost/38029] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:33,069 INFO [Listener at localhost/38029] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:33,069 ERROR [Listener at localhost/38029] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:33,069 INFO [Listener at localhost/38029] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:04:33,069 INFO [Listener at localhost/38029] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:33,069 INFO [Listener at localhost/38029] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:33,069 INFO [Listener at localhost/38029] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:33,070 INFO [Listener at localhost/38029] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:33,071 DEBUG [Listener at localhost/38029] store.ProcedureTree(114): Procedure Procedure(pid=15, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure) stack ids=[0] 2023-07-12 15:04:33,071 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:33,072 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:33,072 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:33,072 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:33,073 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,073 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,073 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(468): Loading pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,073 DEBUG [Listener at localhost/38029] procedure2.Procedure(966): pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:33,074 INFO [Listener at localhost/38029] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 3 msec 2023-07-12 15:04:33,074 DEBUG [Listener at localhost/38029] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:33,075 DEBUG [Listener at localhost/38029] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,44477,1689174267423, table=hbase:meta, region=1588230740 2023-07-12 15:04:33,075 INFO [Listener at localhost/38029] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174303075 2023-07-12 15:04:33,075 DEBUG [Listener at localhost/38029] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:33,076 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,079 DEBUG [Listener at localhost/38029] assignment.RegionStateStore(147): Load hbase:meta entry region=d5915cc2af7dcf8b3ba85977de3a2d50, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,44477,1689174267423, regionLocation=jenkins-hbase5.apache.org,44477,1689174267423, openSeqNum=2 2023-07-12 15:04:33,080 DEBUG [Listener at localhost/38029] assignment.RegionStateStore(147): Load hbase:meta entry region=a2058f5c250e868b28bffe4ac4da36a4, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,44477,1689174267423, regionLocation=jenkins-hbase5.apache.org,44477,1689174267423, openSeqNum=2 2023-07-12 15:04:33,080 INFO [Listener at localhost/38029] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:33,080 INFO [Listener at localhost/38029] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174333080 2023-07-12 15:04:33,080 INFO [Listener at localhost/38029] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174393080 2023-07-12 15:04:33,080 INFO [Listener at localhost/38029] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-07-12 15:04:33,080 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,082 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4 2023-07-12 15:04:33,083 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4 empty. 2023-07-12 15:04:33,083 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.tmp/data/default/testDeleteWithFailoverAtStep4/a2058f5c250e868b28bffe4ac4da36a4 2023-07-12 15:04:33,084 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testDeleteWithFailoverAtStep4 regions 2023-07-12 15:04:33,086 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:04:33,088 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,109 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testDeleteWithFailoverAtStep4 from hbase:meta 2023-07-12 15:04:33,112 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testDeleteWithFailoverAtStep4' descriptor. 2023-07-12 15:04:33,113 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:33,113 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,113 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:33,113 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:33,330 INFO [Listener at localhost/38029] procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,330 INFO [Listener at localhost/38029] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:04:33,330 INFO [Listener at localhost/38029] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:33,330 INFO [Listener at localhost/38029] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:33,331 ERROR [Listener at localhost/38029] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:33,331 INFO [Listener at localhost/38029] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:04:33,331 INFO [Listener at localhost/38029] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:33,331 INFO [Listener at localhost/38029] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:33,331 INFO [Listener at localhost/38029] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:33,332 INFO [Listener at localhost/38029] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:33,333 DEBUG [Listener at localhost/38029] store.ProcedureTree(114): Procedure Procedure(pid=15, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure) stack ids=[0, 1] 2023-07-12 15:04:33,333 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:33,334 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:33,334 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:33,334 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:33,334 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,335 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,335 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(468): Loading pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,335 DEBUG [Listener at localhost/38029] procedure2.Procedure(966): pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:33,335 INFO [Listener at localhost/38029] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 3 msec 2023-07-12 15:04:33,336 DEBUG [Listener at localhost/38029] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:33,336 DEBUG [Listener at localhost/38029] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,44477,1689174267423, table=hbase:meta, region=1588230740 2023-07-12 15:04:33,337 INFO [Listener at localhost/38029] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174303337 2023-07-12 15:04:33,338 DEBUG [Listener at localhost/38029] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:33,338 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,341 DEBUG [Listener at localhost/38029] assignment.RegionStateStore(147): Load hbase:meta entry region=d5915cc2af7dcf8b3ba85977de3a2d50, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,44477,1689174267423, regionLocation=jenkins-hbase5.apache.org,44477,1689174267423, openSeqNum=2 2023-07-12 15:04:33,341 INFO [Listener at localhost/38029] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:33,341 INFO [Listener at localhost/38029] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174333341 2023-07-12 15:04:33,341 INFO [Listener at localhost/38029] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174393341 2023-07-12 15:04:33,341 INFO [Listener at localhost/38029] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 15:04:33,341 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,343 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testDeleteWithFailoverAtStep4' descriptor. 2023-07-12 15:04:33,343 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:04:33,345 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,345 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testDeleteWithFailoverAtStep4' from region states. 2023-07-12 15:04:33,345 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 0 regions from META 2023-07-12 15:04:33,345 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [] 2023-07-12 15:04:33,345 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testDeleteWithFailoverAtStep4' as deleted. 2023-07-12 15:04:33,345 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689174273345"}]},"ts":"9223372036854775807"} 2023-07-12 15:04:33,347 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testDeleteWithFailoverAtStep4 state from META 2023-07-12 15:04:33,349 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:33,349 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,349 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:33,349 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:33,591 INFO [Listener at localhost/38029] procedure.TestMasterFailoverWithProcedures(306): Restart 3 exec state: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,591 INFO [Listener at localhost/38029] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:04:33,591 INFO [Listener at localhost/38029] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:33,591 INFO [Listener at localhost/38029] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:33,592 ERROR [Listener at localhost/38029] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:33,592 INFO [Listener at localhost/38029] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:04:33,592 INFO [Listener at localhost/38029] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:33,592 INFO [Listener at localhost/38029] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:33,592 INFO [Listener at localhost/38029] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:33,593 INFO [Listener at localhost/38029] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:33,594 DEBUG [Listener at localhost/38029] store.ProcedureTree(114): Procedure Procedure(pid=15, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure) stack ids=[0, 1, 2] 2023-07-12 15:04:33,594 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:33,595 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:33,595 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:33,595 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:33,595 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,596 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,596 DEBUG [Listener at localhost/38029] procedure2.ProcedureExecutor(468): Loading pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,596 DEBUG [Listener at localhost/38029] procedure2.Procedure(966): pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:33,596 INFO [Listener at localhost/38029] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 15:04:33,597 DEBUG [Listener at localhost/38029] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:33,597 DEBUG [Listener at localhost/38029] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,44477,1689174267423, table=hbase:meta, region=1588230740 2023-07-12 15:04:33,598 INFO [Listener at localhost/38029] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174303598 2023-07-12 15:04:33,598 DEBUG [Listener at localhost/38029] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:33,599 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,601 DEBUG [Listener at localhost/38029] assignment.RegionStateStore(147): Load hbase:meta entry region=d5915cc2af7dcf8b3ba85977de3a2d50, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,44477,1689174267423, regionLocation=jenkins-hbase5.apache.org,44477,1689174267423, openSeqNum=2 2023-07-12 15:04:33,601 INFO [Listener at localhost/38029] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:33,601 INFO [Listener at localhost/38029] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174333601 2023-07-12 15:04:33,601 INFO [Listener at localhost/38029] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174393601 2023-07-12 15:04:33,601 INFO [Listener at localhost/38029] assignment.AssignmentManager(1536): Joined the cluster in 2 msec 2023-07-12 15:04:33,601 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,601 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testDeleteWithFailoverAtStep4' from region states. 2023-07-12 15:04:33,601 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 0 regions from META 2023-07-12 15:04:33,602 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [] 2023-07-12 15:04:33,602 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testDeleteWithFailoverAtStep4' as deleted. 2023-07-12 15:04:33,602 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689174273602"}]},"ts":"9223372036854775807"} 2023-07-12 15:04:33,604 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testDeleteWithFailoverAtStep4 state from META 2023-07-12 15:04:33,605 DEBUG [PEWorker-1] zookeeper.RecoverableZooKeeper(172): Node /hbase/table/testDeleteWithFailoverAtStep4 already deleted, retry=false 2023-07-12 15:04:33,605 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:04:33,606 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,606 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:33,607 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=15, state=SUCCESS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:33,607 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:33,607 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=15, state=SUCCESS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:33,851 INFO [Listener at localhost/38029] procedure.TestMasterFailoverWithProcedures(313): Trigger master failover 2023-07-12 15:04:33,852 INFO [Listener at localhost/38029] hbase.MiniHBaseCluster(607): Aborting jenkins-hbase5.apache.org,35575,1689174267355 2023-07-12 15:04:33,852 ERROR [Listener at localhost/38029] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 15:04:33,852 ERROR [Listener at localhost/38029] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase5.apache.org,35575,1689174267355: Aborting for tests ***** java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:608) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:411) at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.masterFailover(MasterProcedureTestingUtility.java:135) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testRecoveryAndDoubleExecution(TestMasterFailoverWithProcedures.java:314) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testDeleteWithFailoverAtStep(TestMasterFailoverWithProcedures.java:159) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testDeleteWithFailover(TestMasterFailoverWithProcedures.java:137) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:04:33,852 INFO [Listener at localhost/38029] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,35575,1689174267355' ***** 2023-07-12 15:04:33,852 INFO [Listener at localhost/38029] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost/38029 2023-07-12 15:04:33,853 DEBUG [M:0;jenkins-hbase5:35575] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ca45701, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:04:33,853 INFO [M:0;jenkins-hbase5:35575] regionserver.HRegionServer(1141): aborting server jenkins-hbase5.apache.org,35575,1689174267355 2023-07-12 15:04:33,853 DEBUG [M:0;jenkins-hbase5:35575] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51025d1d to 127.0.0.1:53131 2023-07-12 15:04:33,853 DEBUG [M:0;jenkins-hbase5:35575] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:33,854 INFO [M:0;jenkins-hbase5:35575] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,35575,1689174267355; all regions closed. 2023-07-12 15:04:33,854 DEBUG [M:0;jenkins-hbase5:35575] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:33,854 DEBUG [M:0;jenkins-hbase5:35575] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 15:04:33,854 DEBUG [M:0;jenkins-hbase5:35575] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 15:04:33,854 INFO [M:0;jenkins-hbase5:35575] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 15:04:33,854 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174268780] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174268780,5,FailOnTimeoutGroup] 2023-07-12 15:04:33,855 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 15:04:33,855 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174268780] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174268780,5,FailOnTimeoutGroup] 2023-07-12 15:04:33,854 INFO [M:0;jenkins-hbase5:35575] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 15:04:33,856 INFO [M:0;jenkins-hbase5:35575] hbase.ChoreService(369): Chore service for: master/jenkins-hbase5:0 had [] on shutdown 2023-07-12 15:04:33,856 DEBUG [M:0;jenkins-hbase5:35575] master.HMaster(1512): Stopping service threads 2023-07-12 15:04:33,856 INFO [M:0;jenkins-hbase5:35575] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 15:04:33,856 ERROR [M:0;jenkins-hbase5:35575] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:33,857 INFO [M:0;jenkins-hbase5:35575] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 15:04:33,857 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 15:04:33,857 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:33,857 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:33,857 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:33,857 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): regionserver:44477-0x101c5ac9e960002, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:33,857 DEBUG [M:0;jenkins-hbase5:35575] zookeeper.ZKUtil(398): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 15:04:33,857 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:33,857 WARN [M:0;jenkins-hbase5:35575] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 15:04:33,857 INFO [M:0;jenkins-hbase5:35575] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:33,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:33,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:33,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44477-0x101c5ac9e960002, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:33,859 INFO [M:0;jenkins-hbase5:35575] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 15:04:33,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:33,859 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-12 15:04:33,861 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): regionserver:44477-0x101c5ac9e960002, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:33,861 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:33,861 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:33,861 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:33,861 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:33,862 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/WALs/jenkins-hbase5.apache.org,35575,1689174267355/jenkins-hbase5.apache.org%2C35575%2C1689174267355.1689174268701 not finished, retry = 0 2023-07-12 15:04:33,862 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:33,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:33,863 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase5.apache.org,39873,1689174267392 from backup master directory 2023-07-12 15:04:33,864 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase5.apache.org,39873,1689174267392 2023-07-12 15:04:33,864 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:33,864 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:33,864 WARN [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:04:33,864 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase5.apache.org,39873,1689174267392 2023-07-12 15:04:33,876 INFO [master/jenkins-hbase5:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:33,964 DEBUG [M:0;jenkins-hbase5:35575] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:04:33,964 INFO [M:0;jenkins-hbase5:35575] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:33,964 DEBUG [M:0;jenkins-hbase5:35575] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:33,964 DEBUG [M:0;jenkins-hbase5:35575] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:04:33,964 DEBUG [M:0;jenkins-hbase5:35575] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:33,964 ERROR [M:0;jenkins-hbase5:35575] regionserver.HRegion(1825): Memstore data size is 52357 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:33,964 INFO [M:0;jenkins-hbase5:35575] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:33,965 DEBUG [M:0;jenkins-hbase5:35575] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:33,965 INFO [M:0;jenkins-hbase5:35575] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 15:04:33,965 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:04:33,965 INFO [M:0;jenkins-hbase5:35575] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:35575 2023-07-12 15:04:33,968 DEBUG [M:0;jenkins-hbase5:35575] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase5.apache.org,35575,1689174267355 already deleted, retry=false 2023-07-12 15:04:33,996 WARN [RS-EventLoopGroup-11-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase5.apache.org/172.31.10.131:35575 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:35575 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:04:33,997 DEBUG [RS-EventLoopGroup-11-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase5.apache.org/172.31.10.131:35575 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:35575 2023-07-12 15:04:34,071 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:34,071 INFO [M:0;jenkins-hbase5:35575] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,35575,1689174267355; zookeeper connection closed. 2023-07-12 15:04:34,071 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:35575-0x101c5ac9e960000, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:34,097 DEBUG [RS:0;jenkins-hbase5:44477] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 15:04:34,100 INFO [RS-EventLoopGroup-10-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49951, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 15:04:34,100 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39873] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:34,202 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39873] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:34,291 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x5479dc1f to 127.0.0.1:53131 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:34,295 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@573aa9e4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:34,295 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:34,295 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 15:04:34,296 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:34,302 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/WALs/jenkins-hbase5.apache.org,35575,1689174267355 to hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/WALs/jenkins-hbase5.apache.org,35575,1689174267355-dead as it is dead 2023-07-12 15:04:34,302 INFO [master/jenkins-hbase5:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/WALs/jenkins-hbase5.apache.org,35575,1689174267355-dead/jenkins-hbase5.apache.org%2C35575%2C1689174267355.1689174268701 2023-07-12 15:04:34,303 INFO [master/jenkins-hbase5:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/WALs/jenkins-hbase5.apache.org,35575,1689174267355-dead/jenkins-hbase5.apache.org%2C35575%2C1689174267355.1689174268701 after 1ms 2023-07-12 15:04:34,303 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/WALs/jenkins-hbase5.apache.org,35575,1689174267355-dead/jenkins-hbase5.apache.org%2C35575%2C1689174267355.1689174268701 to hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C35575%2C1689174267355.1689174268701 2023-07-12 15:04:34,303 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/WALs/jenkins-hbase5.apache.org,35575,1689174267355-dead 2023-07-12 15:04:34,304 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/WALs/jenkins-hbase5.apache.org,39873,1689174267392 2023-07-12 15:04:34,305 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39873] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:34,306 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C39873%2C1689174267392, suffix=, logDir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/WALs/jenkins-hbase5.apache.org,39873,1689174267392, archiveDir=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/oldWALs, maxLogs=10 2023-07-12 15:04:34,320 DEBUG [RS-EventLoopGroup-11-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44635,DS-c47add32-855e-45e9-892b-b126e5589b10,DISK] 2023-07-12 15:04:34,321 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/WALs/jenkins-hbase5.apache.org,39873,1689174267392/jenkins-hbase5.apache.org%2C39873%2C1689174267392.1689174274306 2023-07-12 15:04:34,322 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44635,DS-c47add32-855e-45e9-892b-b126e5589b10,DISK]] 2023-07-12 15:04:34,322 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:34,322 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:34,322 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:34,322 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:34,324 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:34,325 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 15:04:34,325 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 15:04:34,326 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:34,327 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-12 15:04:34,327 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C35575%2C1689174267355.1689174268701 2023-07-12 15:04:34,333 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 133, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=119, path=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C35575%2C1689174267355.1689174268701 2023-07-12 15:04:34,333 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=51.13 KB heapSize=61.23 KB 2023-07-12 15:04:34,407 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39873] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:34,509 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39873] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:34,612 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39873] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:34,714 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39873] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:34,742 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.13 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/8bbdf3bdba18404191d19460be426b97 2023-07-12 15:04:34,748 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/8bbdf3bdba18404191d19460be426b97 as hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/8bbdf3bdba18404191d19460be426b97 2023-07-12 15:04:34,753 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/8bbdf3bdba18404191d19460be426b97, entries=133, sequenceid=119, filesize=56.0 K 2023-07-12 15:04:34,753 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~51.13 KB/52357, heapSize ~61.21 KB/62680, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 420ms, sequenceid=119, compaction requested=false; wal=null 2023-07-12 15:04:34,754 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C35575%2C1689174267355.1689174268701 2023-07-12 15:04:34,757 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:34,759 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/119.seqid, newMaxSeqId=119, maxSeqId=1 2023-07-12 15:04:34,760 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=120; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71349387, jitterRate=0.06318871676921844}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:04:34,760 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:34,760 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 15:04:34,761 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:34,761 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:34,761 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:34,762 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:34,766 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] store.ProcedureTree(114): Procedure Procedure(pid=15, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure) stack ids=[0, 1, 2, 3] 2023-07-12 15:04:34,766 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:34,766 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:34,767 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:34,767 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:34,767 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:34,768 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:34,768 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(468): Loading pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:34,768 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.Procedure(966): pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:34,768 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-07-12 15:04:34,768 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 15:04:34,769 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:34,769 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,44477,1689174267423, table=hbase:meta, region=1588230740 2023-07-12 15:04:34,770 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 1 possibly 'live' servers, and 0 'splitting'. 2023-07-12 15:04:34,771 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase5.apache.org,44477,1689174267423 2023-07-12 15:04:34,782 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 15:04:34,782 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 15:04:34,782 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 15:04:34,782 INFO [master/jenkins-hbase5:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 15:04:34,783 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 15:04:34,783 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 15:04:34,784 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 15:04:34,784 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 15:04:34,784 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase5.apache.org,39873,1689174267392, sessionid=0x101c5ac9e960001, setting cluster-up flag (Was=true) 2023-07-12 15:04:34,786 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 15:04:34,787 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,39873,1689174267392 2023-07-12 15:04:34,789 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 15:04:34,790 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,39873,1689174267392 2023-07-12 15:04:34,791 WARN [master/jenkins-hbase5:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/.hbase-snapshot/.tmp 2023-07-12 15:04:34,791 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:34,791 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:34,791 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:34,792 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:34,792 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase5:0, corePoolSize=10, maxPoolSize=10 2023-07-12 15:04:34,792 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:34,792 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:04:34,792 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:34,792 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174304792 2023-07-12 15:04:34,793 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 15:04:34,793 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:34,793 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 15:04:34,793 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 15:04:34,793 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 15:04:34,793 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 15:04:34,793 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 15:04:34,793 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:34,794 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 15:04:34,794 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 15:04:34,794 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 15:04:34,794 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 15:04:34,794 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 15:04:34,794 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174274794,5,FailOnTimeoutGroup] 2023-07-12 15:04:34,794 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174274794,5,FailOnTimeoutGroup] 2023-07-12 15:04:34,794 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:34,794 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 15:04:34,794 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:34,795 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:34,795 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689174274795, completionTime=-1 2023-07-12 15:04:34,795 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 15:04:34,795 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:34,796 DEBUG [hconnection-0x2b6fbf3c-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:04:34,798 INFO [RS-EventLoopGroup-11-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37398, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:04:34,799 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=d5915cc2af7dcf8b3ba85977de3a2d50, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,44477,1689174267423, regionLocation=jenkins-hbase5.apache.org,44477,1689174267423, openSeqNum=2 2023-07-12 15:04:34,799 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:34,799 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174334799 2023-07-12 15:04:34,799 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174394799 2023-07-12 15:04:34,799 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-07-12 15:04:34,807 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,39873,1689174267392-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:34,807 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,39873,1689174267392-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:34,807 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,39873,1689174267392-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:34,807 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase5:39873, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:34,807 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:34,807 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 15:04:34,809 DEBUG [master/jenkins-hbase5:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 15:04:34,819 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 15:04:34,823 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 15:04:34,823 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 0.958sec 2023-07-12 15:04:34,823 INFO [master/jenkins-hbase5:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 15:04:34,823 INFO [master/jenkins-hbase5:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 15:04:34,823 INFO [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 15:04:34,823 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,39873,1689174267392-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 15:04:34,823 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,39873,1689174267392-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 15:04:34,825 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 15:04:34,873 DEBUG [PEWorker-1] procedure.ProcedureSyncWait(216): waitFor quota manager to be available 2023-07-12 15:04:34,873 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 15:04:34,876 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 in 2.3220 sec 2023-07-12 15:04:35,113 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 15:04:35,113 INFO [Listener at localhost/38029] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 15:04:35,113 DEBUG [Listener at localhost/38029] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71f57cde to 127.0.0.1:53131 2023-07-12 15:04:35,113 DEBUG [Listener at localhost/38029] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:35,113 DEBUG [Listener at localhost/38029] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 15:04:35,113 DEBUG [Listener at localhost/38029] util.JVMClusterUtil(257): Found active master hash=882103353, stopped=false 2023-07-12 15:04:35,114 INFO [Listener at localhost/38029] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase5.apache.org,39873,1689174267392 2023-07-12 15:04:35,116 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:35,116 INFO [Listener at localhost/38029] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:35,116 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): regionserver:44477-0x101c5ac9e960002, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:35,116 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:35,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:35,118 DEBUG [Listener at localhost/38029] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5479dc1f to 127.0.0.1:53131 2023-07-12 15:04:35,118 DEBUG [Listener at localhost/38029] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:35,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44477-0x101c5ac9e960002, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:35,119 INFO [Listener at localhost/38029] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,44477,1689174267423' ***** 2023-07-12 15:04:35,119 INFO [Listener at localhost/38029] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 15:04:35,119 INFO [RS:0;jenkins-hbase5:44477] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 15:04:35,119 INFO [RS:0;jenkins-hbase5:44477] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 15:04:35,119 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 15:04:35,119 INFO [RS:0;jenkins-hbase5:44477] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 15:04:35,119 INFO [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(3305): Received CLOSE for d5915cc2af7dcf8b3ba85977de3a2d50 2023-07-12 15:04:35,120 INFO [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,44477,1689174267423 2023-07-12 15:04:35,120 DEBUG [RS:0;jenkins-hbase5:44477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x261edccd to 127.0.0.1:53131 2023-07-12 15:04:35,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing d5915cc2af7dcf8b3ba85977de3a2d50, disabling compactions & flushes 2023-07-12 15:04:35,120 DEBUG [RS:0;jenkins-hbase5:44477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:35,120 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50. 2023-07-12 15:04:35,120 INFO [RS:0;jenkins-hbase5:44477] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 15:04:35,120 INFO [RS:0;jenkins-hbase5:44477] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 15:04:35,120 INFO [RS:0;jenkins-hbase5:44477] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 15:04:35,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50. 2023-07-12 15:04:35,120 INFO [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 15:04:35,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50. after waiting 0 ms 2023-07-12 15:04:35,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50. 2023-07-12 15:04:35,120 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing d5915cc2af7dcf8b3ba85977de3a2d50 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-12 15:04:35,120 INFO [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-07-12 15:04:35,120 DEBUG [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, d5915cc2af7dcf8b3ba85977de3a2d50=hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50.} 2023-07-12 15:04:35,121 DEBUG [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1504): Waiting on 1588230740, d5915cc2af7dcf8b3ba85977de3a2d50 2023-07-12 15:04:35,121 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 15:04:35,121 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 15:04:35,121 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 15:04:35,121 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 15:04:35,121 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 15:04:35,121 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=4.00 KB heapSize=7.39 KB 2023-07-12 15:04:35,184 INFO [regionserver/jenkins-hbase5:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 15:04:35,321 DEBUG [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1504): Waiting on 1588230740, d5915cc2af7dcf8b3ba85977de3a2d50 2023-07-12 15:04:35,457 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-12 15:04:35,521 DEBUG [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1504): Waiting on 1588230740, d5915cc2af7dcf8b3ba85977de3a2d50 2023-07-12 15:04:35,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/namespace/d5915cc2af7dcf8b3ba85977de3a2d50/.tmp/info/0b0603dc14be4fe293cf2c374c158944 2023-07-12 15:04:35,530 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.35 KB at sequenceid=21 (bloomFilter=false), to=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/.tmp/info/b4f7b62071a843f186391eafb138e307 2023-07-12 15:04:35,537 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b4f7b62071a843f186391eafb138e307 2023-07-12 15:04:35,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/namespace/d5915cc2af7dcf8b3ba85977de3a2d50/.tmp/info/0b0603dc14be4fe293cf2c374c158944 as hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/namespace/d5915cc2af7dcf8b3ba85977de3a2d50/info/0b0603dc14be4fe293cf2c374c158944 2023-07-12 15:04:35,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/namespace/d5915cc2af7dcf8b3ba85977de3a2d50/info/0b0603dc14be4fe293cf2c374c158944, entries=2, sequenceid=6, filesize=4.8 K 2023-07-12 15:04:35,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for d5915cc2af7dcf8b3ba85977de3a2d50 in 424ms, sequenceid=6, compaction requested=false 2023-07-12 15:04:35,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/namespace/d5915cc2af7dcf8b3ba85977de3a2d50/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-12 15:04:35,553 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50. 2023-07-12 15:04:35,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for d5915cc2af7dcf8b3ba85977de3a2d50: 2023-07-12 15:04:35,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689174269998.d5915cc2af7dcf8b3ba85977de3a2d50. 2023-07-12 15:04:35,721 DEBUG [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 15:04:35,890 INFO [regionserver/jenkins-hbase5:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-12 15:04:35,890 INFO [regionserver/jenkins-hbase5:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-12 15:04:35,921 DEBUG [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 15:04:35,950 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=109 B at sequenceid=21 (bloomFilter=false), to=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/.tmp/rep_barrier/183c5978908847a6993ce74f3769f9a8 2023-07-12 15:04:35,956 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 183c5978908847a6993ce74f3769f9a8 2023-07-12 15:04:36,122 INFO [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-12 15:04:36,122 DEBUG [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-07-12 15:04:36,122 DEBUG [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 15:04:36,322 DEBUG [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 15:04:36,364 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=559 B at sequenceid=21 (bloomFilter=false), to=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/.tmp/table/efdc243b570e4ff18f578e185a91e6ce 2023-07-12 15:04:36,369 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for efdc243b570e4ff18f578e185a91e6ce 2023-07-12 15:04:36,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/.tmp/info/b4f7b62071a843f186391eafb138e307 as hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/info/b4f7b62071a843f186391eafb138e307 2023-07-12 15:04:36,376 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b4f7b62071a843f186391eafb138e307 2023-07-12 15:04:36,376 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/info/b4f7b62071a843f186391eafb138e307, entries=11, sequenceid=21, filesize=6.2 K 2023-07-12 15:04:36,377 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/.tmp/rep_barrier/183c5978908847a6993ce74f3769f9a8 as hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/rep_barrier/183c5978908847a6993ce74f3769f9a8 2023-07-12 15:04:36,381 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 183c5978908847a6993ce74f3769f9a8 2023-07-12 15:04:36,382 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/rep_barrier/183c5978908847a6993ce74f3769f9a8, entries=1, sequenceid=21, filesize=5.0 K 2023-07-12 15:04:36,382 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/.tmp/table/efdc243b570e4ff18f578e185a91e6ce as hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/table/efdc243b570e4ff18f578e185a91e6ce 2023-07-12 15:04:36,388 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for efdc243b570e4ff18f578e185a91e6ce 2023-07-12 15:04:36,388 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/table/efdc243b570e4ff18f578e185a91e6ce, entries=5, sequenceid=21, filesize=5.1 K 2023-07-12 15:04:36,389 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~4.00 KB/4094, heapSize ~7.34 KB/7520, currentSize=0 B/0 for 1588230740 in 1267ms, sequenceid=21, compaction requested=false 2023-07-12 15:04:36,395 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/data/hbase/meta/1588230740/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=1 2023-07-12 15:04:36,395 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 15:04:36,396 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 15:04:36,396 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 15:04:36,396 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 15:04:36,522 INFO [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,44477,1689174267423; all regions closed. 2023-07-12 15:04:36,528 DEBUG [RS:0;jenkins-hbase5:44477] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/oldWALs 2023-07-12 15:04:36,528 INFO [RS:0;jenkins-hbase5:44477] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C44477%2C1689174267423.meta:.meta(num 1689174269936) 2023-07-12 15:04:36,532 DEBUG [RS:0;jenkins-hbase5:44477] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/oldWALs 2023-07-12 15:04:36,533 INFO [RS:0;jenkins-hbase5:44477] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C44477%2C1689174267423:(num 1689174269005) 2023-07-12 15:04:36,533 DEBUG [RS:0;jenkins-hbase5:44477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:36,533 INFO [RS:0;jenkins-hbase5:44477] regionserver.LeaseManager(133): Closed leases 2023-07-12 15:04:36,533 INFO [RS:0;jenkins-hbase5:44477] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase5:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-12 15:04:36,533 INFO [regionserver/jenkins-hbase5:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:04:36,534 INFO [RS:0;jenkins-hbase5:44477] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:44477 2023-07-12 15:04:36,536 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): regionserver:44477-0x101c5ac9e960002, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase5.apache.org,44477,1689174267423 2023-07-12 15:04:36,536 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:04:36,536 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): regionserver:44477-0x101c5ac9e960002, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:04:36,538 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase5.apache.org,44477,1689174267423] 2023-07-12 15:04:36,538 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase5.apache.org,44477,1689174267423; numProcessing=1 2023-07-12 15:04:36,543 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase5.apache.org,44477,1689174267423 already deleted, retry=false 2023-07-12 15:04:36,543 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase5.apache.org,44477,1689174267423 expired; onlineServers=0 2023-07-12 15:04:36,543 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,39873,1689174267392' ***** 2023-07-12 15:04:36,543 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-12 15:04:36,543 DEBUG [M:1;jenkins-hbase5:39873] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30ac4673, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:04:36,543 INFO [M:1;jenkins-hbase5:39873] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,39873,1689174267392 2023-07-12 15:04:36,543 INFO [M:1;jenkins-hbase5:39873] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,39873,1689174267392; all regions closed. 2023-07-12 15:04:36,543 DEBUG [M:1;jenkins-hbase5:39873] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:36,543 DEBUG [M:1;jenkins-hbase5:39873] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 15:04:36,543 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 15:04:36,543 DEBUG [M:1;jenkins-hbase5:39873] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 15:04:36,543 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174274794] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174274794,5,FailOnTimeoutGroup] 2023-07-12 15:04:36,544 INFO [M:1;jenkins-hbase5:39873] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 15:04:36,544 INFO [M:1;jenkins-hbase5:39873] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 15:04:36,543 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174274794] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174274794,5,FailOnTimeoutGroup] 2023-07-12 15:04:36,544 INFO [M:1;jenkins-hbase5:39873] hbase.ChoreService(369): Chore service for: master/jenkins-hbase5:0 had [] on shutdown 2023-07-12 15:04:36,544 DEBUG [M:1;jenkins-hbase5:39873] master.HMaster(1512): Stopping service threads 2023-07-12 15:04:36,544 INFO [M:1;jenkins-hbase5:39873] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 15:04:36,544 ERROR [M:1;jenkins-hbase5:39873] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:36,545 INFO [M:1;jenkins-hbase5:39873] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:36,545 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 15:04:36,545 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:36,545 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:36,545 DEBUG [M:1;jenkins-hbase5:39873] zookeeper.ZKUtil(398): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 15:04:36,545 WARN [M:1;jenkins-hbase5:39873] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 15:04:36,545 INFO [M:1;jenkins-hbase5:39873] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:36,546 INFO [M:1;jenkins-hbase5:39873] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-12 15:04:36,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:36,546 DEBUG [M:1;jenkins-hbase5:39873] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:04:36,546 INFO [M:1;jenkins-hbase5:39873] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:36,546 DEBUG [M:1;jenkins-hbase5:39873] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:36,546 DEBUG [M:1;jenkins-hbase5:39873] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:04:36,546 DEBUG [M:1;jenkins-hbase5:39873] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:36,546 INFO [M:1;jenkins-hbase5:39873] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=661 B heapSize=1.05 KB 2023-07-12 15:04:36,638 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): regionserver:44477-0x101c5ac9e960002, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:36,638 INFO [RS:0;jenkins-hbase5:44477] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,44477,1689174267423; zookeeper connection closed. 2023-07-12 15:04:36,638 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): regionserver:44477-0x101c5ac9e960002, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:36,640 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@19a062f6] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@19a062f6 2023-07-12 15:04:36,640 INFO [Listener at localhost/38029] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 1 regionserver(s) complete 2023-07-12 15:04:36,955 INFO [M:1;jenkins-hbase5:39873] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=661 B at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d400bb50daca4fe0bea10e5417369dc3 2023-07-12 15:04:36,960 DEBUG [M:1;jenkins-hbase5:39873] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d400bb50daca4fe0bea10e5417369dc3 as hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d400bb50daca4fe0bea10e5417369dc3 2023-07-12 15:04:36,965 INFO [M:1;jenkins-hbase5:39873] regionserver.HStore(1080): Added hdfs://localhost:35443/user/jenkins/test-data/f8ed1f3d-4846-0cf0-c4ea-b0a6e1f29fbb/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d400bb50daca4fe0bea10e5417369dc3, entries=1, sequenceid=123, filesize=5.1 K 2023-07-12 15:04:36,966 INFO [M:1;jenkins-hbase5:39873] regionserver.HRegion(2948): Finished flush of dataSize ~661 B/661, heapSize ~1.03 KB/1056, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 420ms, sequenceid=123, compaction requested=false 2023-07-12 15:04:36,968 INFO [M:1;jenkins-hbase5:39873] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:36,968 DEBUG [M:1;jenkins-hbase5:39873] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:36,971 INFO [M:1;jenkins-hbase5:39873] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 15:04:36,971 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:04:36,972 INFO [M:1;jenkins-hbase5:39873] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:39873 2023-07-12 15:04:36,973 DEBUG [M:1;jenkins-hbase5:39873] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase5.apache.org,39873,1689174267392 already deleted, retry=false 2023-07-12 15:04:37,076 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:37,076 INFO [M:1;jenkins-hbase5:39873] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,39873,1689174267392; zookeeper connection closed. 2023-07-12 15:04:37,076 DEBUG [Listener at localhost/38029-EventThread] zookeeper.ZKWatcher(600): master:39873-0x101c5ac9e960001, quorum=127.0.0.1:53131, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:37,077 WARN [Listener at localhost/38029] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 15:04:37,080 INFO [Listener at localhost/38029] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 15:04:37,184 WARN [BP-369202710-172.31.10.131-1689174266449 heartbeating to localhost/127.0.0.1:35443] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 15:04:37,185 WARN [BP-369202710-172.31.10.131-1689174266449 heartbeating to localhost/127.0.0.1:35443] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-369202710-172.31.10.131-1689174266449 (Datanode Uuid 791d1ad3-98cc-4198-a234-45ded6a5adc6) service to localhost/127.0.0.1:35443 2023-07-12 15:04:37,186 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/cluster_90ed3790-4073-b197-f2f1-bf5abc664ba4/dfs/data/data1/current/BP-369202710-172.31.10.131-1689174266449] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 15:04:37,186 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/cluster_90ed3790-4073-b197-f2f1-bf5abc664ba4/dfs/data/data2/current/BP-369202710-172.31.10.131-1689174266449] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 15:04:37,196 INFO [Listener at localhost/38029] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 15:04:37,305 INFO [Listener at localhost/38029] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 15:04:37,314 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-07-12 15:04:37,325 INFO [Listener at localhost/38029] hbase.ResourceChecker(175): after: master.procedure.TestMasterFailoverWithProcedures#testDeleteWithFailover Thread=102 (was 86) Potentially hanging thread: nioEventLoopGroup-6-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/38029 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (593222724) connection to localhost/127.0.0.1:35443 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: ForkJoinPool-2-worker-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: nioEventLoopGroup-7-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:35443 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:35443 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (593222724) connection to localhost/127.0.0.1:35443 from jenkins.hfs.2 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-11-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (593222724) connection to localhost/127.0.0.1:35443 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-9-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=450 (was 421) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=237 (was 250), ProcessCount=161 (was 161), AvailableMemoryMB=7911 (was 8071) 2023-07-12 15:04:37,333 INFO [Listener at localhost/38029] hbase.ResourceChecker(147): before: master.procedure.TestMasterFailoverWithProcedures#testTruncateWithFailover Thread=102, OpenFileDescriptor=450, MaxFileDescriptor=60000, SystemLoadAverage=237, ProcessCount=161, AvailableMemoryMB=7911 2023-07-12 15:04:37,334 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=2, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-12 15:04:37,334 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/hadoop.log.dir so I do NOT create it in target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232 2023-07-12 15:04:37,334 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8514f246-f21b-9108-ea99-c782f0acda35/hadoop.tmp.dir so I do NOT create it in target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232 2023-07-12 15:04:37,334 INFO [Listener at localhost/38029] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/cluster_a0c31478-57e2-8297-176a-88a31018afde, deleteOnExit=true 2023-07-12 15:04:37,334 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-12 15:04:37,334 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/test.cache.data in system properties and HBase conf 2023-07-12 15:04:37,334 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 15:04:37,334 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/hadoop.log.dir in system properties and HBase conf 2023-07-12 15:04:37,334 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 15:04:37,335 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 15:04:37,335 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 15:04:37,335 DEBUG [Listener at localhost/38029] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 15:04:37,335 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 15:04:37,335 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 15:04:37,335 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 15:04:37,335 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 15:04:37,336 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 15:04:37,336 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 15:04:37,336 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 15:04:37,336 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 15:04:37,336 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 15:04:37,336 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/nfs.dump.dir in system properties and HBase conf 2023-07-12 15:04:37,336 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/java.io.tmpdir in system properties and HBase conf 2023-07-12 15:04:37,336 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 15:04:37,336 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 15:04:37,336 INFO [Listener at localhost/38029] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 15:04:37,341 WARN [Listener at localhost/38029] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 15:04:37,341 WARN [Listener at localhost/38029] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 15:04:37,384 WARN [Listener at localhost/38029] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 15:04:37,386 INFO [Listener at localhost/38029] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 15:04:37,390 INFO [Listener at localhost/38029] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/java.io.tmpdir/Jetty_localhost_39669_hdfs____ryu7lp/webapp 2023-07-12 15:04:37,484 INFO [Listener at localhost/38029] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39669 2023-07-12 15:04:37,489 WARN [Listener at localhost/38029] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 15:04:37,489 WARN [Listener at localhost/38029] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 15:04:37,529 WARN [Listener at localhost/40575] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 15:04:37,540 WARN [Listener at localhost/40575] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 15:04:37,542 WARN [Listener at localhost/40575] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 15:04:37,543 INFO [Listener at localhost/40575] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 15:04:37,547 INFO [Listener at localhost/40575] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/java.io.tmpdir/Jetty_localhost_35025_datanode____.lesonc/webapp 2023-07-12 15:04:37,641 INFO [Listener at localhost/40575] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35025 2023-07-12 15:04:37,647 WARN [Listener at localhost/41275] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 15:04:37,741 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x14353b0c4c7a386d: Processing first storage report for DS-e518b71d-0c70-439e-b158-b43ca736d25c from datanode 2e1fc74d-8826-44de-af2b-d81e3f1a7569 2023-07-12 15:04:37,741 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x14353b0c4c7a386d: from storage DS-e518b71d-0c70-439e-b158-b43ca736d25c node DatanodeRegistration(127.0.0.1:35761, datanodeUuid=2e1fc74d-8826-44de-af2b-d81e3f1a7569, infoPort=33247, infoSecurePort=0, ipcPort=41275, storageInfo=lv=-57;cid=testClusterID;nsid=2124230550;c=1689174277344), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 15:04:37,742 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x14353b0c4c7a386d: Processing first storage report for DS-099223c4-65b1-4b60-bd50-81f7d59146c4 from datanode 2e1fc74d-8826-44de-af2b-d81e3f1a7569 2023-07-12 15:04:37,742 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x14353b0c4c7a386d: from storage DS-099223c4-65b1-4b60-bd50-81f7d59146c4 node DatanodeRegistration(127.0.0.1:35761, datanodeUuid=2e1fc74d-8826-44de-af2b-d81e3f1a7569, infoPort=33247, infoSecurePort=0, ipcPort=41275, storageInfo=lv=-57;cid=testClusterID;nsid=2124230550;c=1689174277344), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 15:04:37,753 DEBUG [Listener at localhost/41275] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232 2023-07-12 15:04:37,755 INFO [Listener at localhost/41275] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/cluster_a0c31478-57e2-8297-176a-88a31018afde/zookeeper_0, clientPort=54427, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/cluster_a0c31478-57e2-8297-176a-88a31018afde/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/cluster_a0c31478-57e2-8297-176a-88a31018afde/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 15:04:37,756 INFO [Listener at localhost/41275] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=54427 2023-07-12 15:04:37,757 INFO [Listener at localhost/41275] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:37,757 INFO [Listener at localhost/41275] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:38,167 INFO [Listener at localhost/41275] util.FSUtils(471): Created version file at hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941 with version=8 2023-07-12 15:04:38,167 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/hbase-staging 2023-07-12 15:04:38,169 INFO [Listener at localhost/41275] client.ConnectionUtils(127): master/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:04:38,169 INFO [Listener at localhost/41275] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:38,169 INFO [Listener at localhost/41275] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:38,169 INFO [Listener at localhost/41275] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:04:38,170 INFO [Listener at localhost/41275] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:38,170 INFO [Listener at localhost/41275] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:04:38,170 INFO [Listener at localhost/41275] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:04:38,171 INFO [Listener at localhost/41275] ipc.NettyRpcServer(120): Bind to /172.31.10.131:37825 2023-07-12 15:04:38,172 INFO [Listener at localhost/41275] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:38,172 INFO [Listener at localhost/41275] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:38,173 INFO [Listener at localhost/41275] zookeeper.RecoverableZooKeeper(93): Process identifier=master:37825 connecting to ZooKeeper ensemble=127.0.0.1:54427 2023-07-12 15:04:38,180 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:378250x0, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:04:38,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:37825-0x101c5acc8d60000 connected 2023-07-12 15:04:38,195 DEBUG [Listener at localhost/41275] zookeeper.ZKUtil(164): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:38,195 DEBUG [Listener at localhost/41275] zookeeper.ZKUtil(164): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:38,195 DEBUG [Listener at localhost/41275] zookeeper.ZKUtil(164): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:04:38,196 DEBUG [Listener at localhost/41275] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37825 2023-07-12 15:04:38,196 DEBUG [Listener at localhost/41275] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37825 2023-07-12 15:04:38,196 DEBUG [Listener at localhost/41275] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37825 2023-07-12 15:04:38,196 DEBUG [Listener at localhost/41275] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37825 2023-07-12 15:04:38,197 DEBUG [Listener at localhost/41275] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37825 2023-07-12 15:04:38,197 INFO [Listener at localhost/41275] master.HMaster(444): hbase.rootdir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941, hbase.cluster.distributed=false 2023-07-12 15:04:38,198 INFO [Listener at localhost/41275] client.ConnectionUtils(127): master/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:04:38,199 INFO [Listener at localhost/41275] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:38,199 INFO [Listener at localhost/41275] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:38,199 INFO [Listener at localhost/41275] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:04:38,199 INFO [Listener at localhost/41275] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:38,199 INFO [Listener at localhost/41275] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:04:38,199 INFO [Listener at localhost/41275] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:04:38,200 INFO [Listener at localhost/41275] ipc.NettyRpcServer(120): Bind to /172.31.10.131:41959 2023-07-12 15:04:38,200 INFO [Listener at localhost/41275] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:38,201 INFO [Listener at localhost/41275] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:38,202 INFO [Listener at localhost/41275] zookeeper.RecoverableZooKeeper(93): Process identifier=master:41959 connecting to ZooKeeper ensemble=127.0.0.1:54427 2023-07-12 15:04:38,205 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:419590x0, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:04:38,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:41959-0x101c5acc8d60001 connected 2023-07-12 15:04:38,208 DEBUG [Listener at localhost/41275] zookeeper.ZKUtil(164): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:38,208 DEBUG [Listener at localhost/41275] zookeeper.ZKUtil(164): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:38,209 DEBUG [Listener at localhost/41275] zookeeper.ZKUtil(164): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:04:38,209 DEBUG [Listener at localhost/41275] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41959 2023-07-12 15:04:38,209 DEBUG [Listener at localhost/41275] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41959 2023-07-12 15:04:38,209 DEBUG [Listener at localhost/41275] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41959 2023-07-12 15:04:38,210 DEBUG [Listener at localhost/41275] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41959 2023-07-12 15:04:38,210 DEBUG [Listener at localhost/41275] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41959 2023-07-12 15:04:38,210 INFO [Listener at localhost/41275] master.HMaster(444): hbase.rootdir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941, hbase.cluster.distributed=false 2023-07-12 15:04:38,223 INFO [Listener at localhost/41275] client.ConnectionUtils(127): regionserver/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:04:38,223 INFO [Listener at localhost/41275] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:38,223 INFO [Listener at localhost/41275] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:38,223 INFO [Listener at localhost/41275] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:04:38,223 INFO [Listener at localhost/41275] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:38,223 INFO [Listener at localhost/41275] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:04:38,223 INFO [Listener at localhost/41275] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:04:38,224 INFO [Listener at localhost/41275] ipc.NettyRpcServer(120): Bind to /172.31.10.131:36649 2023-07-12 15:04:38,224 INFO [Listener at localhost/41275] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 15:04:38,225 DEBUG [Listener at localhost/41275] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 15:04:38,225 INFO [Listener at localhost/41275] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:38,227 INFO [Listener at localhost/41275] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:38,227 INFO [Listener at localhost/41275] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36649 connecting to ZooKeeper ensemble=127.0.0.1:54427 2023-07-12 15:04:38,231 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): regionserver:366490x0, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:04:38,232 DEBUG [Listener at localhost/41275] zookeeper.ZKUtil(164): regionserver:366490x0, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:38,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:36649-0x101c5acc8d60002 connected 2023-07-12 15:04:38,232 DEBUG [Listener at localhost/41275] zookeeper.ZKUtil(164): regionserver:36649-0x101c5acc8d60002, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:38,233 DEBUG [Listener at localhost/41275] zookeeper.ZKUtil(164): regionserver:36649-0x101c5acc8d60002, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:04:38,233 DEBUG [Listener at localhost/41275] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36649 2023-07-12 15:04:38,233 DEBUG [Listener at localhost/41275] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36649 2023-07-12 15:04:38,233 DEBUG [Listener at localhost/41275] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36649 2023-07-12 15:04:38,234 DEBUG [Listener at localhost/41275] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36649 2023-07-12 15:04:38,234 DEBUG [Listener at localhost/41275] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36649 2023-07-12 15:04:38,235 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase5.apache.org,37825,1689174278169 2023-07-12 15:04:38,235 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase5.apache.org,41959,1689174278198 2023-07-12 15:04:38,237 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:38,237 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:38,237 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase5.apache.org,37825,1689174278169 2023-07-12 15:04:38,237 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase5.apache.org,41959,1689174278198 2023-07-12 15:04:38,239 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:38,239 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): regionserver:36649-0x101c5acc8d60002, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:38,239 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:38,239 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:38,239 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:38,240 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:38,240 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:38,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:38,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:38,241 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase5.apache.org,37825,1689174278169 from backup master directory 2023-07-12 15:04:38,241 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase5.apache.org,37825,1689174278169; waiting to become the next active master 2023-07-12 15:04:38,242 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase5.apache.org,37825,1689174278169 2023-07-12 15:04:38,242 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:38,242 WARN [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:04:38,242 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:38,242 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase5.apache.org,37825,1689174278169 2023-07-12 15:04:38,653 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/hbase.id with ID: 7f57b7ef-f220-4531-a143-394481b470c9 2023-07-12 15:04:38,665 INFO [master/jenkins-hbase5:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:38,668 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:38,668 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:39,075 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x393e990b to 127.0.0.1:54427 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:39,078 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e4b601a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:39,079 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:39,079 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 15:04:39,079 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:39,082 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store-tmp 2023-07-12 15:04:39,488 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:39,488 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:04:39,488 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:39,488 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:39,488 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:04:39,488 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:39,488 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:39,488 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:39,489 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/WALs/jenkins-hbase5.apache.org,37825,1689174278169 2023-07-12 15:04:39,492 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C37825%2C1689174278169, suffix=, logDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/WALs/jenkins-hbase5.apache.org,37825,1689174278169, archiveDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/oldWALs, maxLogs=10 2023-07-12 15:04:39,505 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35761,DS-e518b71d-0c70-439e-b158-b43ca736d25c,DISK] 2023-07-12 15:04:39,508 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/WALs/jenkins-hbase5.apache.org,37825,1689174278169/jenkins-hbase5.apache.org%2C37825%2C1689174278169.1689174279492 2023-07-12 15:04:39,508 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35761,DS-e518b71d-0c70-439e-b158-b43ca736d25c,DISK]] 2023-07-12 15:04:39,508 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:39,508 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:39,508 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:39,508 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:39,509 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:39,511 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 15:04:39,511 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 15:04:39,512 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:39,512 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:39,513 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:39,515 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:39,517 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:39,517 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60385483, jitterRate=-0.10018618404865265}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:04:39,517 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:39,517 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 15:04:39,518 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:39,518 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:39,519 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:39,519 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:39,519 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-07-12 15:04:39,519 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 15:04:39,520 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 15:04:39,521 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 15:04:39,531 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 15:04:39,531 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 15:04:39,532 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 15:04:39,532 INFO [master/jenkins-hbase5:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 15:04:39,532 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 15:04:39,534 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:39,534 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:39,534 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 15:04:39,535 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 15:04:39,536 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 15:04:39,537 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:39,537 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:39,537 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:39,537 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): regionserver:36649-0x101c5acc8d60002, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:39,537 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:39,538 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase5.apache.org,37825,1689174278169, sessionid=0x101c5acc8d60000, setting cluster-up flag (Was=false) 2023-07-12 15:04:39,542 INFO [M:1;jenkins-hbase5:41959] regionserver.HRegionServer(951): ClusterId : 7f57b7ef-f220-4531-a143-394481b470c9 2023-07-12 15:04:39,543 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:39,543 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:39,549 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 15:04:39,551 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,37825,1689174278169 2023-07-12 15:04:39,554 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:39,554 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:39,560 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 15:04:39,563 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,37825,1689174278169 2023-07-12 15:04:39,564 WARN [master/jenkins-hbase5:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.hbase-snapshot/.tmp 2023-07-12 15:04:39,566 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:39,567 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:39,567 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:39,567 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:39,567 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:39,567 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase5:0, corePoolSize=10, maxPoolSize=10 2023-07-12 15:04:39,567 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:39,567 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:04:39,567 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:39,571 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174309571 2023-07-12 15:04:39,572 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 15:04:39,572 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 15:04:39,572 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 15:04:39,572 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 15:04:39,572 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 15:04:39,572 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 15:04:39,575 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:39,576 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:39,576 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 15:04:39,577 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 15:04:39,577 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 15:04:39,577 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 15:04:39,577 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 15:04:39,577 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 15:04:39,578 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:39,578 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174279577,5,FailOnTimeoutGroup] 2023-07-12 15:04:39,578 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174279578,5,FailOnTimeoutGroup] 2023-07-12 15:04:39,578 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:39,578 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 15:04:39,578 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:39,578 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:39,637 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(951): ClusterId : 7f57b7ef-f220-4531-a143-394481b470c9 2023-07-12 15:04:39,637 DEBUG [RS:0;jenkins-hbase5:36649] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 15:04:39,639 DEBUG [RS:0;jenkins-hbase5:36649] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 15:04:39,639 DEBUG [RS:0;jenkins-hbase5:36649] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 15:04:39,641 DEBUG [RS:0;jenkins-hbase5:36649] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 15:04:39,642 DEBUG [RS:0;jenkins-hbase5:36649] zookeeper.ReadOnlyZKClient(139): Connect 0x780f8f53 to 127.0.0.1:54427 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:39,647 DEBUG [RS:0;jenkins-hbase5:36649] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76c87e22, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:39,648 DEBUG [RS:0;jenkins-hbase5:36649] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72f7e423, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:04:39,661 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase5:36649 2023-07-12 15:04:39,662 INFO [RS:0;jenkins-hbase5:36649] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 15:04:39,662 INFO [RS:0;jenkins-hbase5:36649] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 15:04:39,662 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 15:04:39,662 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase5.apache.org,37825,1689174278169 with isa=jenkins-hbase5.apache.org/172.31.10.131:36649, startcode=1689174278222 2023-07-12 15:04:39,662 DEBUG [RS:0;jenkins-hbase5:36649] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 15:04:39,666 INFO [RS-EventLoopGroup-12-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:52789, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 15:04:39,667 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=37825] master.ServerManager(394): Registering regionserver=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:39,667 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941 2023-07-12 15:04:39,667 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:40575 2023-07-12 15:04:39,668 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 15:04:39,669 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:04:39,670 DEBUG [RS:0;jenkins-hbase5:36649] zookeeper.ZKUtil(162): regionserver:36649-0x101c5acc8d60002, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:39,670 WARN [RS:0;jenkins-hbase5:36649] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:04:39,670 INFO [RS:0;jenkins-hbase5:36649] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:39,670 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase5.apache.org,36649,1689174278222] 2023-07-12 15:04:39,670 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1948): logDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/WALs/jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:39,675 DEBUG [RS:0;jenkins-hbase5:36649] zookeeper.ZKUtil(162): regionserver:36649-0x101c5acc8d60002, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:39,676 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 15:04:39,676 INFO [RS:0;jenkins-hbase5:36649] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 15:04:39,679 INFO [RS:0;jenkins-hbase5:36649] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 15:04:39,679 INFO [RS:0;jenkins-hbase5:36649] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:04:39,680 INFO [RS:0;jenkins-hbase5:36649] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:39,680 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 15:04:39,681 INFO [RS:0;jenkins-hbase5:36649] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:39,681 DEBUG [RS:0;jenkins-hbase5:36649] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:39,681 DEBUG [RS:0;jenkins-hbase5:36649] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:39,681 DEBUG [RS:0;jenkins-hbase5:36649] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:39,682 DEBUG [RS:0;jenkins-hbase5:36649] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:39,682 DEBUG [RS:0;jenkins-hbase5:36649] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:39,682 DEBUG [RS:0;jenkins-hbase5:36649] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:04:39,682 DEBUG [RS:0;jenkins-hbase5:36649] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:39,682 DEBUG [RS:0;jenkins-hbase5:36649] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:39,682 DEBUG [RS:0;jenkins-hbase5:36649] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:39,682 DEBUG [RS:0;jenkins-hbase5:36649] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:39,683 INFO [RS:0;jenkins-hbase5:36649] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:39,683 INFO [RS:0;jenkins-hbase5:36649] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:39,683 INFO [RS:0;jenkins-hbase5:36649] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:39,695 INFO [RS:0;jenkins-hbase5:36649] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 15:04:39,695 INFO [RS:0;jenkins-hbase5:36649] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,36649,1689174278222-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:39,707 INFO [RS:0;jenkins-hbase5:36649] regionserver.Replication(203): jenkins-hbase5.apache.org,36649,1689174278222 started 2023-07-12 15:04:39,707 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1637): Serving as jenkins-hbase5.apache.org,36649,1689174278222, RpcServer on jenkins-hbase5.apache.org/172.31.10.131:36649, sessionid=0x101c5acc8d60002 2023-07-12 15:04:39,707 DEBUG [RS:0;jenkins-hbase5:36649] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 15:04:39,707 DEBUG [RS:0;jenkins-hbase5:36649] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:39,707 DEBUG [RS:0;jenkins-hbase5:36649] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,36649,1689174278222' 2023-07-12 15:04:39,707 DEBUG [RS:0;jenkins-hbase5:36649] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:04:39,707 DEBUG [RS:0;jenkins-hbase5:36649] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:04:39,708 DEBUG [RS:0;jenkins-hbase5:36649] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 15:04:39,708 DEBUG [RS:0;jenkins-hbase5:36649] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 15:04:39,708 DEBUG [RS:0;jenkins-hbase5:36649] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:39,708 DEBUG [RS:0;jenkins-hbase5:36649] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,36649,1689174278222' 2023-07-12 15:04:39,708 DEBUG [RS:0;jenkins-hbase5:36649] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 15:04:39,708 DEBUG [RS:0;jenkins-hbase5:36649] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 15:04:39,708 DEBUG [RS:0;jenkins-hbase5:36649] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 15:04:39,708 INFO [RS:0;jenkins-hbase5:36649] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 15:04:39,708 INFO [RS:0;jenkins-hbase5:36649] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 15:04:39,811 INFO [RS:0;jenkins-hbase5:36649] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C36649%2C1689174278222, suffix=, logDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/WALs/jenkins-hbase5.apache.org,36649,1689174278222, archiveDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/oldWALs, maxLogs=32 2023-07-12 15:04:39,826 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35761,DS-e518b71d-0c70-439e-b158-b43ca736d25c,DISK] 2023-07-12 15:04:39,828 INFO [RS:0;jenkins-hbase5:36649] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/WALs/jenkins-hbase5.apache.org,36649,1689174278222/jenkins-hbase5.apache.org%2C36649%2C1689174278222.1689174279812 2023-07-12 15:04:39,829 DEBUG [RS:0;jenkins-hbase5:36649] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35761,DS-e518b71d-0c70-439e-b158-b43ca736d25c,DISK]] 2023-07-12 15:04:39,990 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:39,990 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:39,990 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941 2023-07-12 15:04:40,397 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:40,398 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 15:04:40,400 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/info 2023-07-12 15:04:40,400 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 15:04:40,400 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:40,401 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 15:04:40,402 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:04:40,402 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 15:04:40,403 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:40,403 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 15:04:40,404 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/table 2023-07-12 15:04:40,404 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 15:04:40,404 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:40,405 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740 2023-07-12 15:04:40,405 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740 2023-07-12 15:04:40,408 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 15:04:40,409 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 15:04:40,411 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:40,411 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60443231, jitterRate=-0.09932567179203033}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 15:04:40,411 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 15:04:40,412 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 15:04:40,412 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 15:04:40,412 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 15:04:40,412 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 15:04:40,412 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 15:04:40,412 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 15:04:40,412 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 15:04:40,413 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:40,413 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 15:04:40,413 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 15:04:40,414 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 15:04:40,415 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 15:04:40,565 DEBUG [jenkins-hbase5:37825] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 15:04:40,566 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,36649,1689174278222, state=OPENING 2023-07-12 15:04:40,568 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 15:04:40,569 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:40,569 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:40,569 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,36649,1689174278222}] 2023-07-12 15:04:40,569 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:04:40,569 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:04:40,722 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:40,722 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 15:04:40,725 INFO [RS-EventLoopGroup-14-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:46108, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 15:04:40,728 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 15:04:40,728 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:40,730 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C36649%2C1689174278222.meta, suffix=.meta, logDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/WALs/jenkins-hbase5.apache.org,36649,1689174278222, archiveDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/oldWALs, maxLogs=32 2023-07-12 15:04:40,745 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35761,DS-e518b71d-0c70-439e-b158-b43ca736d25c,DISK] 2023-07-12 15:04:40,747 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/WALs/jenkins-hbase5.apache.org,36649,1689174278222/jenkins-hbase5.apache.org%2C36649%2C1689174278222.meta.1689174280731.meta 2023-07-12 15:04:40,747 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35761,DS-e518b71d-0c70-439e-b158-b43ca736d25c,DISK]] 2023-07-12 15:04:40,748 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:40,748 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 15:04:40,748 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 15:04:40,748 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 15:04:40,748 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 15:04:40,748 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:40,748 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 15:04:40,748 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 15:04:40,749 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 15:04:40,751 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/info 2023-07-12 15:04:40,751 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/info 2023-07-12 15:04:40,751 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 15:04:40,752 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:40,752 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 15:04:40,753 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:04:40,753 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:04:40,753 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 15:04:40,754 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:40,754 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 15:04:40,755 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/table 2023-07-12 15:04:40,755 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/table 2023-07-12 15:04:40,755 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 15:04:40,756 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:40,757 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740 2023-07-12 15:04:40,758 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740 2023-07-12 15:04:40,760 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 15:04:40,761 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 15:04:40,762 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63223731, jitterRate=-0.057892993092536926}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 15:04:40,762 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 15:04:40,763 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689174280722 2023-07-12 15:04:40,766 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 15:04:40,766 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 15:04:40,766 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,36649,1689174278222, state=OPEN 2023-07-12 15:04:40,773 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 15:04:40,773 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 15:04:40,773 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:04:40,773 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:04:40,775 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 15:04:40,775 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,36649,1689174278222 in 204 msec 2023-07-12 15:04:40,776 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 15:04:40,777 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 362 msec 2023-07-12 15:04:40,778 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.2120 sec 2023-07-12 15:04:40,779 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689174280779, completionTime=-1 2023-07-12 15:04:40,779 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 15:04:40,779 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:40,781 DEBUG [hconnection-0x4dff6c87-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:04:40,783 INFO [RS-EventLoopGroup-14-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:46120, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:04:40,784 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:40,785 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174340785 2023-07-12 15:04:40,785 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174400785 2023-07-12 15:04:40,785 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-07-12 15:04:40,790 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,37825,1689174278169-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:40,790 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,37825,1689174278169-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:40,790 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,37825,1689174278169-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:40,790 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase5:37825, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:40,790 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:40,790 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 15:04:40,790 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:40,791 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:40,791 DEBUG [master/jenkins-hbase5:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 15:04:40,792 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:04:40,793 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:04:40,794 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/hbase/namespace/2f8aded98651ba93c1387ac5ba7e99cf 2023-07-12 15:04:40,795 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/hbase/namespace/2f8aded98651ba93c1387ac5ba7e99cf empty. 2023-07-12 15:04:40,795 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/hbase/namespace/2f8aded98651ba93c1387ac5ba7e99cf 2023-07-12 15:04:40,795 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 15:04:41,054 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-12 15:04:41,204 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:41,205 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2f8aded98651ba93c1387ac5ba7e99cf, NAME => 'hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp 2023-07-12 15:04:41,611 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:41,611 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 2f8aded98651ba93c1387ac5ba7e99cf, disabling compactions & flushes 2023-07-12 15:04:41,611 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf. 2023-07-12 15:04:41,611 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf. 2023-07-12 15:04:41,611 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf. after waiting 0 ms 2023-07-12 15:04:41,611 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf. 2023-07-12 15:04:41,611 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf. 2023-07-12 15:04:41,611 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 2f8aded98651ba93c1387ac5ba7e99cf: 2023-07-12 15:04:41,614 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:04:41,615 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689174281615"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174281615"}]},"ts":"1689174281615"} 2023-07-12 15:04:41,617 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 15:04:41,618 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:04:41,618 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174281618"}]},"ts":"1689174281618"} 2023-07-12 15:04:41,620 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 15:04:41,623 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=2f8aded98651ba93c1387ac5ba7e99cf, ASSIGN}] 2023-07-12 15:04:41,625 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=2f8aded98651ba93c1387ac5ba7e99cf, ASSIGN 2023-07-12 15:04:41,626 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=2f8aded98651ba93c1387ac5ba7e99cf, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,36649,1689174278222; forceNewPlan=false, retain=false 2023-07-12 15:04:41,776 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=2f8aded98651ba93c1387ac5ba7e99cf, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:41,777 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689174281776"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174281776"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174281776"}]},"ts":"1689174281776"} 2023-07-12 15:04:41,779 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 2f8aded98651ba93c1387ac5ba7e99cf, server=jenkins-hbase5.apache.org,36649,1689174278222}] 2023-07-12 15:04:41,936 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf. 2023-07-12 15:04:41,936 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2f8aded98651ba93c1387ac5ba7e99cf, NAME => 'hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:41,936 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 2f8aded98651ba93c1387ac5ba7e99cf 2023-07-12 15:04:41,936 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:41,936 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 2f8aded98651ba93c1387ac5ba7e99cf 2023-07-12 15:04:41,936 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 2f8aded98651ba93c1387ac5ba7e99cf 2023-07-12 15:04:41,937 INFO [StoreOpener-2f8aded98651ba93c1387ac5ba7e99cf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 2f8aded98651ba93c1387ac5ba7e99cf 2023-07-12 15:04:41,939 DEBUG [StoreOpener-2f8aded98651ba93c1387ac5ba7e99cf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/namespace/2f8aded98651ba93c1387ac5ba7e99cf/info 2023-07-12 15:04:41,939 DEBUG [StoreOpener-2f8aded98651ba93c1387ac5ba7e99cf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/namespace/2f8aded98651ba93c1387ac5ba7e99cf/info 2023-07-12 15:04:41,939 INFO [StoreOpener-2f8aded98651ba93c1387ac5ba7e99cf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2f8aded98651ba93c1387ac5ba7e99cf columnFamilyName info 2023-07-12 15:04:41,940 INFO [StoreOpener-2f8aded98651ba93c1387ac5ba7e99cf-1] regionserver.HStore(310): Store=2f8aded98651ba93c1387ac5ba7e99cf/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:41,941 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/namespace/2f8aded98651ba93c1387ac5ba7e99cf 2023-07-12 15:04:41,941 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/namespace/2f8aded98651ba93c1387ac5ba7e99cf 2023-07-12 15:04:41,944 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 2f8aded98651ba93c1387ac5ba7e99cf 2023-07-12 15:04:41,946 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/namespace/2f8aded98651ba93c1387ac5ba7e99cf/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:41,946 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 2f8aded98651ba93c1387ac5ba7e99cf; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63677700, jitterRate=-0.0511283278465271}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:04:41,946 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 2f8aded98651ba93c1387ac5ba7e99cf: 2023-07-12 15:04:41,947 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf., pid=6, masterSystemTime=1689174281931 2023-07-12 15:04:41,950 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf. 2023-07-12 15:04:41,950 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf. 2023-07-12 15:04:41,950 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=2f8aded98651ba93c1387ac5ba7e99cf, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:41,950 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689174281950"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174281950"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174281950"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174281950"}]},"ts":"1689174281950"} 2023-07-12 15:04:41,953 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 15:04:41,953 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 2f8aded98651ba93c1387ac5ba7e99cf, server=jenkins-hbase5.apache.org,36649,1689174278222 in 173 msec 2023-07-12 15:04:41,955 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 15:04:41,955 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=2f8aded98651ba93c1387ac5ba7e99cf, ASSIGN in 331 msec 2023-07-12 15:04:41,956 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 15:04:41,956 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174281956"}]},"ts":"1689174281956"} 2023-07-12 15:04:41,957 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 15:04:41,960 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 15:04:41,961 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.1700 sec 2023-07-12 15:04:41,993 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 15:04:41,994 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:04:41,994 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:41,994 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:41,998 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:42,005 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:04:42,009 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 11 msec 2023-07-12 15:04:42,020 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:42,027 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:04:42,029 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 9 msec 2023-07-12 15:04:42,035 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 15:04:42,037 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 15:04:42,037 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.794sec 2023-07-12 15:04:42,037 INFO [master/jenkins-hbase5:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 15:04:42,037 INFO [master/jenkins-hbase5:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 15:04:42,037 INFO [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 15:04:42,037 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,37825,1689174278169-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 15:04:42,038 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,37825,1689174278169-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 15:04:42,038 DEBUG [Listener at localhost/41275] zookeeper.ReadOnlyZKClient(139): Connect 0x7c281a1e to 127.0.0.1:54427 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:42,041 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 15:04:42,044 DEBUG [Listener at localhost/41275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a14a29c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:42,045 DEBUG [hconnection-0x67d7341e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:04:42,047 INFO [RS-EventLoopGroup-14-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:46122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:04:42,049 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase5.apache.org,37825,1689174278169 2023-07-12 15:04:42,049 WARN [Listener at localhost/41275] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-12 15:04:42,051 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 15:04:42,053 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:04:42,054 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:04:42,058 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5 2023-07-12 15:04:42,058 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0 2023-07-12 15:04:42,058 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884 2023-07-12 15:04:42,058 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a 2023-07-12 15:04:42,059 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0 empty. 2023-07-12 15:04:42,059 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5 empty. 2023-07-12 15:04:42,060 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a empty. 2023-07-12 15:04:42,060 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884 empty. 2023-07-12 15:04:42,060 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0 2023-07-12 15:04:42,060 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5 2023-07-12 15:04:42,060 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a 2023-07-12 15:04:42,060 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884 2023-07-12 15:04:42,060 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTruncateWithFailoverAtStep4 regions 2023-07-12 15:04:42,470 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:42,471 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(7675): creating {ENCODED => 89dfad7a04faa4508cda27c8a2180884, NAME => 'testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884.', STARTKEY => 'b', ENDKEY => 'c'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp 2023-07-12 15:04:42,472 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(7675): creating {ENCODED => 717d918e7432e9c5e806f8190b78f43a, NAME => 'testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a.', STARTKEY => 'a', ENDKEY => 'b'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp 2023-07-12 15:04:42,472 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(7675): creating {ENCODED => a317889067b8b7fcd34d1e178c37dcb5, NAME => 'testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5.', STARTKEY => '', ENDKEY => 'a'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp 2023-07-12 15:04:42,484 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:42,484 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1604): Closing 717d918e7432e9c5e806f8190b78f43a, disabling compactions & flushes 2023-07-12 15:04:42,484 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a. 2023-07-12 15:04:42,484 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a. 2023-07-12 15:04:42,484 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a. after waiting 0 ms 2023-07-12 15:04:42,484 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a. 2023-07-12 15:04:42,484 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a. 2023-07-12 15:04:42,484 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1558): Region close journal for 717d918e7432e9c5e806f8190b78f43a: 2023-07-12 15:04:42,485 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(7675): creating {ENCODED => faa8bcd2b476118ae57d0b34a73eb8f0, NAME => 'testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0.', STARTKEY => 'c', ENDKEY => ''}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp 2023-07-12 15:04:42,492 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:42,492 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1604): Closing faa8bcd2b476118ae57d0b34a73eb8f0, disabling compactions & flushes 2023-07-12 15:04:42,492 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0. 2023-07-12 15:04:42,492 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0. 2023-07-12 15:04:42,492 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0. after waiting 0 ms 2023-07-12 15:04:42,492 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0. 2023-07-12 15:04:42,492 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0. 2023-07-12 15:04:42,492 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1558): Region close journal for faa8bcd2b476118ae57d0b34a73eb8f0: 2023-07-12 15:04:42,885 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:42,885 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1604): Closing a317889067b8b7fcd34d1e178c37dcb5, disabling compactions & flushes 2023-07-12 15:04:42,885 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5. 2023-07-12 15:04:42,885 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5. 2023-07-12 15:04:42,885 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5. after waiting 0 ms 2023-07-12 15:04:42,885 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5. 2023-07-12 15:04:42,885 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5. 2023-07-12 15:04:42,885 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1558): Region close journal for a317889067b8b7fcd34d1e178c37dcb5: 2023-07-12 15:04:42,885 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:42,885 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1604): Closing 89dfad7a04faa4508cda27c8a2180884, disabling compactions & flushes 2023-07-12 15:04:42,885 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884. 2023-07-12 15:04:42,885 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884. 2023-07-12 15:04:42,885 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884. after waiting 0 ms 2023-07-12 15:04:42,885 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884. 2023-07-12 15:04:42,885 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884. 2023-07-12 15:04:42,886 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1558): Region close journal for 89dfad7a04faa4508cda27c8a2180884: 2023-07-12 15:04:42,888 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:04:42,889 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174282889"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174282889"}]},"ts":"1689174282889"} 2023-07-12 15:04:42,889 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174282889"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174282889"}]},"ts":"1689174282889"} 2023-07-12 15:04:42,889 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174282889"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174282889"}]},"ts":"1689174282889"} 2023-07-12 15:04:42,889 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174282889"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174282889"}]},"ts":"1689174282889"} 2023-07-12 15:04:42,893 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 4 regions to meta. 2023-07-12 15:04:42,894 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:04:42,894 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174282894"}]},"ts":"1689174282894"} 2023-07-12 15:04:42,895 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=ENABLING in hbase:meta 2023-07-12 15:04:42,900 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=a317889067b8b7fcd34d1e178c37dcb5, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=717d918e7432e9c5e806f8190b78f43a, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=89dfad7a04faa4508cda27c8a2180884, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=faa8bcd2b476118ae57d0b34a73eb8f0, ASSIGN}] 2023-07-12 15:04:42,902 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=faa8bcd2b476118ae57d0b34a73eb8f0, ASSIGN 2023-07-12 15:04:42,902 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=faa8bcd2b476118ae57d0b34a73eb8f0, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,36649,1689174278222; forceNewPlan=false, retain=false 2023-07-12 15:04:42,903 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=89dfad7a04faa4508cda27c8a2180884, ASSIGN 2023-07-12 15:04:42,903 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=89dfad7a04faa4508cda27c8a2180884, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,36649,1689174278222; forceNewPlan=false, retain=false 2023-07-12 15:04:42,904 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=717d918e7432e9c5e806f8190b78f43a, ASSIGN 2023-07-12 15:04:42,905 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=717d918e7432e9c5e806f8190b78f43a, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,36649,1689174278222; forceNewPlan=false, retain=false 2023-07-12 15:04:42,905 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=a317889067b8b7fcd34d1e178c37dcb5, ASSIGN 2023-07-12 15:04:42,906 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=a317889067b8b7fcd34d1e178c37dcb5, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,36649,1689174278222; forceNewPlan=false, retain=false 2023-07-12 15:04:43,054 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=717d918e7432e9c5e806f8190b78f43a, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:43,055 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174283054"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174283054"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174283054"}]},"ts":"1689174283054"} 2023-07-12 15:04:43,057 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=11, state=RUNNABLE; OpenRegionProcedure 717d918e7432e9c5e806f8190b78f43a, server=jenkins-hbase5.apache.org,36649,1689174278222}] 2023-07-12 15:04:43,060 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=a317889067b8b7fcd34d1e178c37dcb5, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:43,060 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174283060"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174283060"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174283060"}]},"ts":"1689174283060"} 2023-07-12 15:04:43,062 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=10, state=RUNNABLE; OpenRegionProcedure a317889067b8b7fcd34d1e178c37dcb5, server=jenkins-hbase5.apache.org,36649,1689174278222}] 2023-07-12 15:04:43,065 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=89dfad7a04faa4508cda27c8a2180884, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:43,065 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174283065"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174283065"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174283065"}]},"ts":"1689174283065"} 2023-07-12 15:04:43,066 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=12, state=RUNNABLE; OpenRegionProcedure 89dfad7a04faa4508cda27c8a2180884, server=jenkins-hbase5.apache.org,36649,1689174278222}] 2023-07-12 15:04:43,069 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=faa8bcd2b476118ae57d0b34a73eb8f0, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:43,069 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174283069"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174283069"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174283069"}]},"ts":"1689174283069"} 2023-07-12 15:04:43,071 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=13, state=RUNNABLE; OpenRegionProcedure faa8bcd2b476118ae57d0b34a73eb8f0, server=jenkins-hbase5.apache.org,36649,1689174278222}] 2023-07-12 15:04:43,213 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a. 2023-07-12 15:04:43,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 717d918e7432e9c5e806f8190b78f43a, NAME => 'testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a.', STARTKEY => 'a', ENDKEY => 'b'} 2023-07-12 15:04:43,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 717d918e7432e9c5e806f8190b78f43a 2023-07-12 15:04:43,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:43,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 717d918e7432e9c5e806f8190b78f43a 2023-07-12 15:04:43,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 717d918e7432e9c5e806f8190b78f43a 2023-07-12 15:04:43,215 INFO [StoreOpener-717d918e7432e9c5e806f8190b78f43a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 717d918e7432e9c5e806f8190b78f43a 2023-07-12 15:04:43,217 DEBUG [StoreOpener-717d918e7432e9c5e806f8190b78f43a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/f1 2023-07-12 15:04:43,217 DEBUG [StoreOpener-717d918e7432e9c5e806f8190b78f43a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/f1 2023-07-12 15:04:43,217 INFO [StoreOpener-717d918e7432e9c5e806f8190b78f43a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 717d918e7432e9c5e806f8190b78f43a columnFamilyName f1 2023-07-12 15:04:43,217 INFO [StoreOpener-717d918e7432e9c5e806f8190b78f43a-1] regionserver.HStore(310): Store=717d918e7432e9c5e806f8190b78f43a/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:43,218 INFO [StoreOpener-717d918e7432e9c5e806f8190b78f43a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 717d918e7432e9c5e806f8190b78f43a 2023-07-12 15:04:43,219 DEBUG [StoreOpener-717d918e7432e9c5e806f8190b78f43a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/f2 2023-07-12 15:04:43,219 DEBUG [StoreOpener-717d918e7432e9c5e806f8190b78f43a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/f2 2023-07-12 15:04:43,219 INFO [StoreOpener-717d918e7432e9c5e806f8190b78f43a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 717d918e7432e9c5e806f8190b78f43a columnFamilyName f2 2023-07-12 15:04:43,220 INFO [StoreOpener-717d918e7432e9c5e806f8190b78f43a-1] regionserver.HStore(310): Store=717d918e7432e9c5e806f8190b78f43a/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:43,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a 2023-07-12 15:04:43,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a 2023-07-12 15:04:43,223 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:04:43,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 717d918e7432e9c5e806f8190b78f43a 2023-07-12 15:04:43,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:43,228 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 717d918e7432e9c5e806f8190b78f43a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67197127, jitterRate=0.0013152211904525757}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:04:43,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 717d918e7432e9c5e806f8190b78f43a: 2023-07-12 15:04:43,229 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a., pid=14, masterSystemTime=1689174283209 2023-07-12 15:04:43,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a. 2023-07-12 15:04:43,231 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a. 2023-07-12 15:04:43,231 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5. 2023-07-12 15:04:43,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a317889067b8b7fcd34d1e178c37dcb5, NAME => 'testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5.', STARTKEY => '', ENDKEY => 'a'} 2023-07-12 15:04:43,231 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=717d918e7432e9c5e806f8190b78f43a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:43,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 a317889067b8b7fcd34d1e178c37dcb5 2023-07-12 15:04:43,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:43,231 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174283231"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174283231"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174283231"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174283231"}]},"ts":"1689174283231"} 2023-07-12 15:04:43,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for a317889067b8b7fcd34d1e178c37dcb5 2023-07-12 15:04:43,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for a317889067b8b7fcd34d1e178c37dcb5 2023-07-12 15:04:43,233 INFO [StoreOpener-a317889067b8b7fcd34d1e178c37dcb5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region a317889067b8b7fcd34d1e178c37dcb5 2023-07-12 15:04:43,234 DEBUG [StoreOpener-a317889067b8b7fcd34d1e178c37dcb5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/f1 2023-07-12 15:04:43,234 DEBUG [StoreOpener-a317889067b8b7fcd34d1e178c37dcb5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/f1 2023-07-12 15:04:43,234 INFO [StoreOpener-a317889067b8b7fcd34d1e178c37dcb5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a317889067b8b7fcd34d1e178c37dcb5 columnFamilyName f1 2023-07-12 15:04:43,235 INFO [StoreOpener-a317889067b8b7fcd34d1e178c37dcb5-1] regionserver.HStore(310): Store=a317889067b8b7fcd34d1e178c37dcb5/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:43,235 INFO [StoreOpener-a317889067b8b7fcd34d1e178c37dcb5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region a317889067b8b7fcd34d1e178c37dcb5 2023-07-12 15:04:43,235 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=11 2023-07-12 15:04:43,236 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=11, state=SUCCESS; OpenRegionProcedure 717d918e7432e9c5e806f8190b78f43a, server=jenkins-hbase5.apache.org,36649,1689174278222 in 177 msec 2023-07-12 15:04:43,236 DEBUG [StoreOpener-a317889067b8b7fcd34d1e178c37dcb5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/f2 2023-07-12 15:04:43,236 DEBUG [StoreOpener-a317889067b8b7fcd34d1e178c37dcb5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/f2 2023-07-12 15:04:43,237 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=717d918e7432e9c5e806f8190b78f43a, ASSIGN in 336 msec 2023-07-12 15:04:43,237 INFO [StoreOpener-a317889067b8b7fcd34d1e178c37dcb5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a317889067b8b7fcd34d1e178c37dcb5 columnFamilyName f2 2023-07-12 15:04:43,237 INFO [StoreOpener-a317889067b8b7fcd34d1e178c37dcb5-1] regionserver.HStore(310): Store=a317889067b8b7fcd34d1e178c37dcb5/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:43,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5 2023-07-12 15:04:43,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5 2023-07-12 15:04:43,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:04:43,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for a317889067b8b7fcd34d1e178c37dcb5 2023-07-12 15:04:43,242 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:43,242 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened a317889067b8b7fcd34d1e178c37dcb5; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59161578, jitterRate=-0.11842378973960876}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:04:43,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for a317889067b8b7fcd34d1e178c37dcb5: 2023-07-12 15:04:43,243 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5., pid=15, masterSystemTime=1689174283209 2023-07-12 15:04:43,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5. 2023-07-12 15:04:43,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5. 2023-07-12 15:04:43,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884. 2023-07-12 15:04:43,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 89dfad7a04faa4508cda27c8a2180884, NAME => 'testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884.', STARTKEY => 'b', ENDKEY => 'c'} 2023-07-12 15:04:43,245 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=a317889067b8b7fcd34d1e178c37dcb5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:43,245 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174283245"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174283245"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174283245"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174283245"}]},"ts":"1689174283245"} 2023-07-12 15:04:43,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 89dfad7a04faa4508cda27c8a2180884 2023-07-12 15:04:43,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:43,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 89dfad7a04faa4508cda27c8a2180884 2023-07-12 15:04:43,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 89dfad7a04faa4508cda27c8a2180884 2023-07-12 15:04:43,246 INFO [StoreOpener-89dfad7a04faa4508cda27c8a2180884-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 89dfad7a04faa4508cda27c8a2180884 2023-07-12 15:04:43,248 DEBUG [StoreOpener-89dfad7a04faa4508cda27c8a2180884-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/f1 2023-07-12 15:04:43,248 DEBUG [StoreOpener-89dfad7a04faa4508cda27c8a2180884-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/f1 2023-07-12 15:04:43,248 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=10 2023-07-12 15:04:43,248 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=10, state=SUCCESS; OpenRegionProcedure a317889067b8b7fcd34d1e178c37dcb5, server=jenkins-hbase5.apache.org,36649,1689174278222 in 185 msec 2023-07-12 15:04:43,248 INFO [StoreOpener-89dfad7a04faa4508cda27c8a2180884-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 89dfad7a04faa4508cda27c8a2180884 columnFamilyName f1 2023-07-12 15:04:43,248 INFO [StoreOpener-89dfad7a04faa4508cda27c8a2180884-1] regionserver.HStore(310): Store=89dfad7a04faa4508cda27c8a2180884/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:43,249 INFO [StoreOpener-89dfad7a04faa4508cda27c8a2180884-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 89dfad7a04faa4508cda27c8a2180884 2023-07-12 15:04:43,249 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=a317889067b8b7fcd34d1e178c37dcb5, ASSIGN in 348 msec 2023-07-12 15:04:43,250 DEBUG [StoreOpener-89dfad7a04faa4508cda27c8a2180884-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/f2 2023-07-12 15:04:43,250 DEBUG [StoreOpener-89dfad7a04faa4508cda27c8a2180884-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/f2 2023-07-12 15:04:43,250 INFO [StoreOpener-89dfad7a04faa4508cda27c8a2180884-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 89dfad7a04faa4508cda27c8a2180884 columnFamilyName f2 2023-07-12 15:04:43,250 INFO [StoreOpener-89dfad7a04faa4508cda27c8a2180884-1] regionserver.HStore(310): Store=89dfad7a04faa4508cda27c8a2180884/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:43,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884 2023-07-12 15:04:43,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884 2023-07-12 15:04:43,253 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:04:43,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 89dfad7a04faa4508cda27c8a2180884 2023-07-12 15:04:43,255 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:43,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 89dfad7a04faa4508cda27c8a2180884; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72502559, jitterRate=0.08037231862545013}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:04:43,256 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 89dfad7a04faa4508cda27c8a2180884: 2023-07-12 15:04:43,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884., pid=16, masterSystemTime=1689174283209 2023-07-12 15:04:43,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884. 2023-07-12 15:04:43,258 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884. 2023-07-12 15:04:43,258 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0. 2023-07-12 15:04:43,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => faa8bcd2b476118ae57d0b34a73eb8f0, NAME => 'testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0.', STARTKEY => 'c', ENDKEY => ''} 2023-07-12 15:04:43,258 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=89dfad7a04faa4508cda27c8a2180884, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:43,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 faa8bcd2b476118ae57d0b34a73eb8f0 2023-07-12 15:04:43,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:43,258 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174283258"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174283258"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174283258"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174283258"}]},"ts":"1689174283258"} 2023-07-12 15:04:43,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for faa8bcd2b476118ae57d0b34a73eb8f0 2023-07-12 15:04:43,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for faa8bcd2b476118ae57d0b34a73eb8f0 2023-07-12 15:04:43,259 INFO [StoreOpener-faa8bcd2b476118ae57d0b34a73eb8f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region faa8bcd2b476118ae57d0b34a73eb8f0 2023-07-12 15:04:43,261 DEBUG [StoreOpener-faa8bcd2b476118ae57d0b34a73eb8f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/f1 2023-07-12 15:04:43,261 DEBUG [StoreOpener-faa8bcd2b476118ae57d0b34a73eb8f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/f1 2023-07-12 15:04:43,261 INFO [StoreOpener-faa8bcd2b476118ae57d0b34a73eb8f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region faa8bcd2b476118ae57d0b34a73eb8f0 columnFamilyName f1 2023-07-12 15:04:43,261 INFO [StoreOpener-faa8bcd2b476118ae57d0b34a73eb8f0-1] regionserver.HStore(310): Store=faa8bcd2b476118ae57d0b34a73eb8f0/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:43,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=12 2023-07-12 15:04:43,262 INFO [StoreOpener-faa8bcd2b476118ae57d0b34a73eb8f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region faa8bcd2b476118ae57d0b34a73eb8f0 2023-07-12 15:04:43,262 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=12, state=SUCCESS; OpenRegionProcedure 89dfad7a04faa4508cda27c8a2180884, server=jenkins-hbase5.apache.org,36649,1689174278222 in 194 msec 2023-07-12 15:04:43,263 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=89dfad7a04faa4508cda27c8a2180884, ASSIGN in 362 msec 2023-07-12 15:04:43,263 DEBUG [StoreOpener-faa8bcd2b476118ae57d0b34a73eb8f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/f2 2023-07-12 15:04:43,263 DEBUG [StoreOpener-faa8bcd2b476118ae57d0b34a73eb8f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/f2 2023-07-12 15:04:43,263 INFO [StoreOpener-faa8bcd2b476118ae57d0b34a73eb8f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region faa8bcd2b476118ae57d0b34a73eb8f0 columnFamilyName f2 2023-07-12 15:04:43,264 INFO [StoreOpener-faa8bcd2b476118ae57d0b34a73eb8f0-1] regionserver.HStore(310): Store=faa8bcd2b476118ae57d0b34a73eb8f0/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:43,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0 2023-07-12 15:04:43,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0 2023-07-12 15:04:43,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:04:43,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for faa8bcd2b476118ae57d0b34a73eb8f0 2023-07-12 15:04:43,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:43,270 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened faa8bcd2b476118ae57d0b34a73eb8f0; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69402663, jitterRate=0.03418026864528656}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:04:43,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for faa8bcd2b476118ae57d0b34a73eb8f0: 2023-07-12 15:04:43,271 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0., pid=17, masterSystemTime=1689174283209 2023-07-12 15:04:43,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0. 2023-07-12 15:04:43,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0. 2023-07-12 15:04:43,273 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=faa8bcd2b476118ae57d0b34a73eb8f0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:43,273 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174283273"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174283273"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174283273"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174283273"}]},"ts":"1689174283273"} 2023-07-12 15:04:43,276 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=13 2023-07-12 15:04:43,276 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=13, state=SUCCESS; OpenRegionProcedure faa8bcd2b476118ae57d0b34a73eb8f0, server=jenkins-hbase5.apache.org,36649,1689174278222 in 204 msec 2023-07-12 15:04:43,278 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=9 2023-07-12 15:04:43,278 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=faa8bcd2b476118ae57d0b34a73eb8f0, ASSIGN in 376 msec 2023-07-12 15:04:43,278 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 15:04:43,279 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174283279"}]},"ts":"1689174283279"} 2023-07-12 15:04:43,280 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=ENABLED in hbase:meta 2023-07-12 15:04:43,283 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 15:04:43,285 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 in 1.2330 sec 2023-07-12 15:04:43,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36649] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5. with WAL disabled. Data may be lost in the event of a crash. 2023-07-12 15:04:43,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36649] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a. with WAL disabled. Data may be lost in the event of a crash. 2023-07-12 15:04:43,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36649] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884. with WAL disabled. Data may be lost in the event of a crash. 2023-07-12 15:04:43,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36649] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0. with WAL disabled. Data may be lost in the event of a crash. 2023-07-12 15:04:43,376 DEBUG [Listener at localhost/41275] client.ClientScanner(240): Advancing internal scanner to startKey at 'a', inclusive 2023-07-12 15:04:43,378 DEBUG [Listener at localhost/41275] client.ClientScanner(240): Advancing internal scanner to startKey at 'b', inclusive 2023-07-12 15:04:43,379 DEBUG [Listener at localhost/41275] client.ClientScanner(240): Advancing internal scanner to startKey at 'c', inclusive 2023-07-12 15:04:43,382 DEBUG [Listener at localhost/41275] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 15:04:43,384 INFO [RS-EventLoopGroup-12-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:56470, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 15:04:43,384 INFO [Listener at localhost/41275] client.HBaseAdmin$15(890): Started disable of testTruncateWithFailoverAtStep4 2023-07-12 15:04:43,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37825] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testTruncateWithFailoverAtStep4 2023-07-12 15:04:43,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37825] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 15:04:43,388 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174283388"}]},"ts":"1689174283388"} 2023-07-12 15:04:43,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37825] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 15:04:43,389 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=DISABLING in hbase:meta 2023-07-12 15:04:43,391 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testTruncateWithFailoverAtStep4 to state=DISABLING 2023-07-12 15:04:43,392 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=a317889067b8b7fcd34d1e178c37dcb5, UNASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=717d918e7432e9c5e806f8190b78f43a, UNASSIGN}, {pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=89dfad7a04faa4508cda27c8a2180884, UNASSIGN}, {pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=faa8bcd2b476118ae57d0b34a73eb8f0, UNASSIGN}] 2023-07-12 15:04:43,392 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=faa8bcd2b476118ae57d0b34a73eb8f0, UNASSIGN 2023-07-12 15:04:43,393 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=faa8bcd2b476118ae57d0b34a73eb8f0, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:43,393 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174283393"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174283393"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174283393"}]},"ts":"1689174283393"} 2023-07-12 15:04:43,394 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; CloseRegionProcedure faa8bcd2b476118ae57d0b34a73eb8f0, server=jenkins-hbase5.apache.org,36649,1689174278222}] 2023-07-12 15:04:43,396 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=89dfad7a04faa4508cda27c8a2180884, UNASSIGN 2023-07-12 15:04:43,397 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=89dfad7a04faa4508cda27c8a2180884, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:43,397 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174283397"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174283397"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174283397"}]},"ts":"1689174283397"} 2023-07-12 15:04:43,398 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=21, state=RUNNABLE; CloseRegionProcedure 89dfad7a04faa4508cda27c8a2180884, server=jenkins-hbase5.apache.org,36649,1689174278222}] 2023-07-12 15:04:43,400 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=717d918e7432e9c5e806f8190b78f43a, UNASSIGN 2023-07-12 15:04:43,400 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=717d918e7432e9c5e806f8190b78f43a, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:43,400 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174283400"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174283400"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174283400"}]},"ts":"1689174283400"} 2023-07-12 15:04:43,402 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=20, state=RUNNABLE; CloseRegionProcedure 717d918e7432e9c5e806f8190b78f43a, server=jenkins-hbase5.apache.org,36649,1689174278222}] 2023-07-12 15:04:43,404 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=a317889067b8b7fcd34d1e178c37dcb5, UNASSIGN 2023-07-12 15:04:43,404 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=a317889067b8b7fcd34d1e178c37dcb5, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:43,405 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174283404"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174283404"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174283404"}]},"ts":"1689174283404"} 2023-07-12 15:04:43,406 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=19, state=RUNNABLE; CloseRegionProcedure a317889067b8b7fcd34d1e178c37dcb5, server=jenkins-hbase5.apache.org,36649,1689174278222}] 2023-07-12 15:04:43,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37825] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 15:04:43,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close faa8bcd2b476118ae57d0b34a73eb8f0 2023-07-12 15:04:43,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing faa8bcd2b476118ae57d0b34a73eb8f0, disabling compactions & flushes 2023-07-12 15:04:43,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0. 2023-07-12 15:04:43,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0. 2023-07-12 15:04:43,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0. after waiting 0 ms 2023-07-12 15:04:43,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0. 2023-07-12 15:04:43,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing faa8bcd2b476118ae57d0b34a73eb8f0 2/2 column families, dataSize=3.79 KB heapSize=8.66 KB 2023-07-12 15:04:43,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37825] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 15:04:43,956 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.90 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/.tmp/f1/93592056952e4250bbd281b9f8dd4f2c 2023-07-12 15:04:43,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37825] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 15:04:44,369 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.90 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/.tmp/f2/aa73c2b700744db9aa643c0276871857 2023-07-12 15:04:44,376 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/.tmp/f1/93592056952e4250bbd281b9f8dd4f2c as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/f1/93592056952e4250bbd281b9f8dd4f2c 2023-07-12 15:04:44,381 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/f1/93592056952e4250bbd281b9f8dd4f2c, entries=29, sequenceid=5, filesize=6.8 K 2023-07-12 15:04:44,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/.tmp/f2/aa73c2b700744db9aa643c0276871857 as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/f2/aa73c2b700744db9aa643c0276871857 2023-07-12 15:04:44,387 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/f2/aa73c2b700744db9aa643c0276871857, entries=29, sequenceid=5, filesize=6.8 K 2023-07-12 15:04:44,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.79 KB/3882, heapSize ~8.63 KB/8832, currentSize=0 B/0 for faa8bcd2b476118ae57d0b34a73eb8f0 in 841ms, sequenceid=5, compaction requested=false 2023-07-12 15:04:44,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testTruncateWithFailoverAtStep4' 2023-07-12 15:04:44,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 15:04:44,394 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0. 2023-07-12 15:04:44,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for faa8bcd2b476118ae57d0b34a73eb8f0: 2023-07-12 15:04:44,396 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed faa8bcd2b476118ae57d0b34a73eb8f0 2023-07-12 15:04:44,396 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 89dfad7a04faa4508cda27c8a2180884 2023-07-12 15:04:44,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 89dfad7a04faa4508cda27c8a2180884, disabling compactions & flushes 2023-07-12 15:04:44,398 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884. 2023-07-12 15:04:44,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884. 2023-07-12 15:04:44,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884. after waiting 0 ms 2023-07-12 15:04:44,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884. 2023-07-12 15:04:44,398 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 89dfad7a04faa4508cda27c8a2180884 2/2 column families, dataSize=934 B heapSize=2.47 KB 2023-07-12 15:04:44,398 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=faa8bcd2b476118ae57d0b34a73eb8f0, regionState=CLOSED 2023-07-12 15:04:44,398 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174284398"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174284398"}]},"ts":"1689174284398"} 2023-07-12 15:04:44,401 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-12 15:04:44,401 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; CloseRegionProcedure faa8bcd2b476118ae57d0b34a73eb8f0, server=jenkins-hbase5.apache.org,36649,1689174278222 in 1.0060 sec 2023-07-12 15:04:44,403 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=faa8bcd2b476118ae57d0b34a73eb8f0, UNASSIGN in 1.0100 sec 2023-07-12 15:04:44,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37825] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 15:04:44,806 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=467 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/.tmp/f1/4083942195e44107b3c934c4c223452d 2023-07-12 15:04:45,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=467 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/.tmp/f2/c74ebec95fb24404abeb7e8af880a551 2023-07-12 15:04:45,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/.tmp/f1/4083942195e44107b3c934c4c223452d as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/f1/4083942195e44107b3c934c4c223452d 2023-07-12 15:04:45,231 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/f1/4083942195e44107b3c934c4c223452d, entries=7, sequenceid=5, filesize=5.3 K 2023-07-12 15:04:45,232 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/.tmp/f2/c74ebec95fb24404abeb7e8af880a551 as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/f2/c74ebec95fb24404abeb7e8af880a551 2023-07-12 15:04:45,237 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/f2/c74ebec95fb24404abeb7e8af880a551, entries=7, sequenceid=5, filesize=5.3 K 2023-07-12 15:04:45,238 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~934 B/934, heapSize ~2.44 KB/2496, currentSize=0 B/0 for 89dfad7a04faa4508cda27c8a2180884 in 840ms, sequenceid=5, compaction requested=false 2023-07-12 15:04:45,246 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 15:04:45,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884. 2023-07-12 15:04:45,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 89dfad7a04faa4508cda27c8a2180884: 2023-07-12 15:04:45,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 89dfad7a04faa4508cda27c8a2180884 2023-07-12 15:04:45,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close a317889067b8b7fcd34d1e178c37dcb5 2023-07-12 15:04:45,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing a317889067b8b7fcd34d1e178c37dcb5, disabling compactions & flushes 2023-07-12 15:04:45,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5. 2023-07-12 15:04:45,249 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=89dfad7a04faa4508cda27c8a2180884, regionState=CLOSED 2023-07-12 15:04:45,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5. 2023-07-12 15:04:45,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5. after waiting 0 ms 2023-07-12 15:04:45,249 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174285249"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174285249"}]},"ts":"1689174285249"} 2023-07-12 15:04:45,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5. 2023-07-12 15:04:45,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing a317889067b8b7fcd34d1e178c37dcb5 2/2 column families, dataSize=7.59 KB heapSize=16.81 KB 2023-07-12 15:04:45,252 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=21 2023-07-12 15:04:45,252 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=21, state=SUCCESS; CloseRegionProcedure 89dfad7a04faa4508cda27c8a2180884, server=jenkins-hbase5.apache.org,36649,1689174278222 in 1.8530 sec 2023-07-12 15:04:45,253 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=89dfad7a04faa4508cda27c8a2180884, UNASSIGN in 1.8610 sec 2023-07-12 15:04:45,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37825] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 15:04:45,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.79 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/.tmp/f1/5b6e41bfe31e4f429f2fe505ddd5a1cb 2023-07-12 15:04:45,676 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 15:04:45,676 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 15:04:46,074 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.79 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/.tmp/f2/005ff515f4254045891582ded0e86612 2023-07-12 15:04:46,081 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/.tmp/f1/5b6e41bfe31e4f429f2fe505ddd5a1cb as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/f1/5b6e41bfe31e4f429f2fe505ddd5a1cb 2023-07-12 15:04:46,087 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/f1/5b6e41bfe31e4f429f2fe505ddd5a1cb, entries=58, sequenceid=5, filesize=8.8 K 2023-07-12 15:04:46,089 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/.tmp/f2/005ff515f4254045891582ded0e86612 as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/f2/005ff515f4254045891582ded0e86612 2023-07-12 15:04:46,095 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/f2/005ff515f4254045891582ded0e86612, entries=58, sequenceid=5, filesize=8.8 K 2023-07-12 15:04:46,096 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~7.59 KB/7772, heapSize ~16.78 KB/17184, currentSize=0 B/0 for a317889067b8b7fcd34d1e178c37dcb5 in 847ms, sequenceid=5, compaction requested=false 2023-07-12 15:04:46,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 15:04:46,103 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5. 2023-07-12 15:04:46,104 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for a317889067b8b7fcd34d1e178c37dcb5: 2023-07-12 15:04:46,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed a317889067b8b7fcd34d1e178c37dcb5 2023-07-12 15:04:46,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 717d918e7432e9c5e806f8190b78f43a 2023-07-12 15:04:46,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 717d918e7432e9c5e806f8190b78f43a, disabling compactions & flushes 2023-07-12 15:04:46,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a. 2023-07-12 15:04:46,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a. 2023-07-12 15:04:46,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a. after waiting 0 ms 2023-07-12 15:04:46,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a. 2023-07-12 15:04:46,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 717d918e7432e9c5e806f8190b78f43a 2/2 column families, dataSize=800 B heapSize=2.19 KB 2023-07-12 15:04:46,106 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=a317889067b8b7fcd34d1e178c37dcb5, regionState=CLOSED 2023-07-12 15:04:46,106 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174286106"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174286106"}]},"ts":"1689174286106"} 2023-07-12 15:04:46,109 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=19 2023-07-12 15:04:46,109 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=19, state=SUCCESS; CloseRegionProcedure a317889067b8b7fcd34d1e178c37dcb5, server=jenkins-hbase5.apache.org,36649,1689174278222 in 2.7020 sec 2023-07-12 15:04:46,110 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=a317889067b8b7fcd34d1e178c37dcb5, UNASSIGN in 2.7180 sec 2023-07-12 15:04:46,117 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=400 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/.tmp/f1/b03b71ec56ce481999418287ee550070 2023-07-12 15:04:46,250 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-12 15:04:46,530 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=400 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/.tmp/f2/654559bae6464767b98b29e6dbf30f9e 2023-07-12 15:04:46,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/.tmp/f1/b03b71ec56ce481999418287ee550070 as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/f1/b03b71ec56ce481999418287ee550070 2023-07-12 15:04:46,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/f1/b03b71ec56ce481999418287ee550070, entries=6, sequenceid=5, filesize=5.2 K 2023-07-12 15:04:46,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/.tmp/f2/654559bae6464767b98b29e6dbf30f9e as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/f2/654559bae6464767b98b29e6dbf30f9e 2023-07-12 15:04:46,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/f2/654559bae6464767b98b29e6dbf30f9e, entries=6, sequenceid=5, filesize=5.2 K 2023-07-12 15:04:46,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~800 B/800, heapSize ~2.16 KB/2208, currentSize=0 B/0 for 717d918e7432e9c5e806f8190b78f43a in 444ms, sequenceid=5, compaction requested=false 2023-07-12 15:04:46,556 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 15:04:46,557 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a. 2023-07-12 15:04:46,557 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 717d918e7432e9c5e806f8190b78f43a: 2023-07-12 15:04:46,559 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 717d918e7432e9c5e806f8190b78f43a 2023-07-12 15:04:46,559 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=717d918e7432e9c5e806f8190b78f43a, regionState=CLOSED 2023-07-12 15:04:46,559 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174286559"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174286559"}]},"ts":"1689174286559"} 2023-07-12 15:04:46,561 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=20 2023-07-12 15:04:46,562 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=20, state=SUCCESS; CloseRegionProcedure 717d918e7432e9c5e806f8190b78f43a, server=jenkins-hbase5.apache.org,36649,1689174278222 in 3.1580 sec 2023-07-12 15:04:46,563 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=18 2023-07-12 15:04:46,563 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=717d918e7432e9c5e806f8190b78f43a, UNASSIGN in 3.1710 sec 2023-07-12 15:04:46,563 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174286563"}]},"ts":"1689174286563"} 2023-07-12 15:04:46,564 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=DISABLED in hbase:meta 2023-07-12 15:04:46,566 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testTruncateWithFailoverAtStep4 to state=DISABLED 2023-07-12 15:04:46,568 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 in 3.1820 sec 2023-07-12 15:04:47,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37825] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 15:04:47,497 INFO [Listener at localhost/41275] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testTruncateWithFailoverAtStep4, procId: 18 completed 2023-07-12 15:04:47,497 WARN [Listener at localhost/41275] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-07-12 15:04:47,504 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:TRUNCATE_TABLE_PRE_OPERATION; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:47,505 DEBUG [PEWorker-1] procedure.TruncateTableProcedure(87): waiting for 'testTruncateWithFailoverAtStep4' regions in transition 2023-07-12 15:04:47,506 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:47,506 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:47,506 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:47,506 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:47,754 INFO [Listener at localhost/41275] procedure.TestMasterFailoverWithProcedures(306): Restart 0 exec state: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:47,754 INFO [Listener at localhost/41275] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:04:47,754 INFO [Listener at localhost/41275] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:47,754 INFO [Listener at localhost/41275] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:47,756 ERROR [Listener at localhost/41275] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:47,757 INFO [Listener at localhost/41275] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:04:47,757 INFO [Listener at localhost/41275] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:47,757 INFO [Listener at localhost/41275] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:47,757 INFO [Listener at localhost/41275] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:47,758 INFO [Listener at localhost/41275] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:47,759 DEBUG [Listener at localhost/41275] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure) stack ids=[] 2023-07-12 15:04:47,759 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:47,759 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:47,760 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:47,760 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:47,760 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 15:04:47,760 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 15:04:47,760 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:TRUNCATE_TABLE_PRE_OPERATION; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:47,761 DEBUG [Listener at localhost/41275] procedure2.Procedure(966): pid=27, state=RUNNABLE:TRUNCATE_TABLE_PRE_OPERATION; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:47,761 INFO [Listener at localhost/41275] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 15:04:47,761 DEBUG [Listener at localhost/41275] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:47,762 DEBUG [Listener at localhost/41275] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,36649,1689174278222, table=hbase:meta, region=1588230740 2023-07-12 15:04:47,762 INFO [Listener at localhost/41275] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174317762 2023-07-12 15:04:47,763 DEBUG [Listener at localhost/41275] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:47,763 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:TRUNCATE_TABLE_PRE_OPERATION; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:47,766 DEBUG [Listener at localhost/41275] assignment.RegionStateStore(147): Load hbase:meta entry region=2f8aded98651ba93c1387ac5ba7e99cf, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,36649,1689174278222, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222, openSeqNum=2 2023-07-12 15:04:47,766 DEBUG [Listener at localhost/41275] assignment.RegionStateStore(147): Load hbase:meta entry region=a317889067b8b7fcd34d1e178c37dcb5, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,36649,1689174278222, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222, openSeqNum=2 2023-07-12 15:04:47,766 DEBUG [Listener at localhost/41275] assignment.RegionStateStore(147): Load hbase:meta entry region=717d918e7432e9c5e806f8190b78f43a, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,36649,1689174278222, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222, openSeqNum=2 2023-07-12 15:04:47,766 DEBUG [Listener at localhost/41275] assignment.RegionStateStore(147): Load hbase:meta entry region=89dfad7a04faa4508cda27c8a2180884, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,36649,1689174278222, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222, openSeqNum=2 2023-07-12 15:04:47,767 DEBUG [Listener at localhost/41275] assignment.RegionStateStore(147): Load hbase:meta entry region=faa8bcd2b476118ae57d0b34a73eb8f0, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,36649,1689174278222, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222, openSeqNum=2 2023-07-12 15:04:47,767 INFO [Listener at localhost/41275] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:47,767 INFO [Listener at localhost/41275] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174347767 2023-07-12 15:04:47,767 INFO [Listener at localhost/41275] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174407767 2023-07-12 15:04:47,767 INFO [Listener at localhost/41275] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 15:04:47,767 DEBUG [PEWorker-1] procedure.TruncateTableProcedure(87): waiting for 'testTruncateWithFailoverAtStep4' regions in transition 2023-07-12 15:04:47,767 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:04:47,775 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5 2023-07-12 15:04:47,775 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0 2023-07-12 15:04:47,775 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a 2023-07-12 15:04:47,775 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884 2023-07-12 15:04:47,778 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/f1, FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/f2, FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/recovered.edits] 2023-07-12 15:04:47,778 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/f1, FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/f2, FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/recovered.edits] 2023-07-12 15:04:47,779 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/f1, FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/f2, FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/recovered.edits] 2023-07-12 15:04:47,779 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/f1, FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/f2, FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/recovered.edits] 2023-07-12 15:04:47,783 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/f1/93592056952e4250bbd281b9f8dd4f2c to hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/archive/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/f1/93592056952e4250bbd281b9f8dd4f2c 2023-07-12 15:04:47,785 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/f1/4083942195e44107b3c934c4c223452d to hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/archive/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/f1/4083942195e44107b3c934c4c223452d 2023-07-12 15:04:47,785 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/f1/5b6e41bfe31e4f429f2fe505ddd5a1cb to hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/archive/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/f1/5b6e41bfe31e4f429f2fe505ddd5a1cb 2023-07-12 15:04:47,786 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/f1/b03b71ec56ce481999418287ee550070 to hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/archive/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/f1/b03b71ec56ce481999418287ee550070 2023-07-12 15:04:47,787 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 15:04:47,788 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-12 15:04:47,789 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/f2/aa73c2b700744db9aa643c0276871857 to hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/archive/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/f2/aa73c2b700744db9aa643c0276871857 2023-07-12 15:04:47,790 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/f2/c74ebec95fb24404abeb7e8af880a551 to hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/archive/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/f2/c74ebec95fb24404abeb7e8af880a551 2023-07-12 15:04:47,791 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/f2/654559bae6464767b98b29e6dbf30f9e to hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/archive/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/f2/654559bae6464767b98b29e6dbf30f9e 2023-07-12 15:04:47,791 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/f2/005ff515f4254045891582ded0e86612 to hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/archive/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/f2/005ff515f4254045891582ded0e86612 2023-07-12 15:04:47,794 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/recovered.edits/8.seqid to hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/archive/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0/recovered.edits/8.seqid 2023-07-12 15:04:47,795 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0 2023-07-12 15:04:47,796 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/recovered.edits/8.seqid to hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/archive/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a/recovered.edits/8.seqid 2023-07-12 15:04:47,796 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/recovered.edits/8.seqid to hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/archive/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884/recovered.edits/8.seqid 2023-07-12 15:04:47,796 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/recovered.edits/8.seqid to hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/archive/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5/recovered.edits/8.seqid 2023-07-12 15:04:47,796 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a 2023-07-12 15:04:47,797 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884 2023-07-12 15:04:47,797 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5 2023-07-12 15:04:47,797 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTruncateWithFailoverAtStep4 regions 2023-07-12 15:04:47,798 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:47,798 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_REMOVE_FROM_META, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:47,798 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:47,798 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_REMOVE_FROM_META, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:48,017 INFO [Listener at localhost/41275] procedure.TestMasterFailoverWithProcedures(306): Restart 1 exec state: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:48,017 INFO [Listener at localhost/41275] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:04:48,017 INFO [Listener at localhost/41275] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:48,017 INFO [Listener at localhost/41275] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:48,018 ERROR [Listener at localhost/41275] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:48,018 INFO [Listener at localhost/41275] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:04:48,018 INFO [Listener at localhost/41275] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:48,018 INFO [Listener at localhost/41275] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:48,018 INFO [Listener at localhost/41275] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:48,019 INFO [Listener at localhost/41275] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:48,020 DEBUG [Listener at localhost/41275] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure) stack ids=[0] 2023-07-12 15:04:48,020 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:48,020 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:48,021 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:48,021 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:48,021 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 15:04:48,022 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 15:04:48,022 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:48,022 DEBUG [Listener at localhost/41275] procedure2.Procedure(966): pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:48,023 INFO [Listener at localhost/41275] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 3 msec 2023-07-12 15:04:48,023 DEBUG [Listener at localhost/41275] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:48,024 DEBUG [Listener at localhost/41275] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,36649,1689174278222, table=hbase:meta, region=1588230740 2023-07-12 15:04:48,024 INFO [Listener at localhost/41275] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174318024 2023-07-12 15:04:48,024 DEBUG [Listener at localhost/41275] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:48,025 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:48,028 DEBUG [Listener at localhost/41275] assignment.RegionStateStore(147): Load hbase:meta entry region=2f8aded98651ba93c1387ac5ba7e99cf, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,36649,1689174278222, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222, openSeqNum=2 2023-07-12 15:04:48,028 DEBUG [Listener at localhost/41275] assignment.RegionStateStore(147): Load hbase:meta entry region=a317889067b8b7fcd34d1e178c37dcb5, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,36649,1689174278222, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222, openSeqNum=2 2023-07-12 15:04:48,028 DEBUG [Listener at localhost/41275] assignment.RegionStateStore(147): Load hbase:meta entry region=717d918e7432e9c5e806f8190b78f43a, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,36649,1689174278222, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222, openSeqNum=2 2023-07-12 15:04:48,028 DEBUG [Listener at localhost/41275] assignment.RegionStateStore(147): Load hbase:meta entry region=89dfad7a04faa4508cda27c8a2180884, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,36649,1689174278222, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222, openSeqNum=2 2023-07-12 15:04:48,028 DEBUG [Listener at localhost/41275] assignment.RegionStateStore(147): Load hbase:meta entry region=faa8bcd2b476118ae57d0b34a73eb8f0, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,36649,1689174278222, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222, openSeqNum=2 2023-07-12 15:04:48,028 INFO [Listener at localhost/41275] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:48,028 INFO [Listener at localhost/41275] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174348028 2023-07-12 15:04:48,028 INFO [Listener at localhost/41275] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174408028 2023-07-12 15:04:48,029 INFO [Listener at localhost/41275] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-07-12 15:04:48,033 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5 2023-07-12 15:04:48,033 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0 2023-07-12 15:04:48,033 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884 2023-07-12 15:04:48,033 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a 2023-07-12 15:04:48,033 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5 empty. 2023-07-12 15:04:48,034 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884 empty. 2023-07-12 15:04:48,034 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0 empty. 2023-07-12 15:04:48,034 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a empty. 2023-07-12 15:04:48,034 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/a317889067b8b7fcd34d1e178c37dcb5 2023-07-12 15:04:48,034 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/89dfad7a04faa4508cda27c8a2180884 2023-07-12 15:04:48,034 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/faa8bcd2b476118ae57d0b34a73eb8f0 2023-07-12 15:04:48,034 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/717d918e7432e9c5e806f8190b78f43a 2023-07-12 15:04:48,034 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTruncateWithFailoverAtStep4 regions 2023-07-12 15:04:48,036 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:04:48,040 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 4 rows of testTruncateWithFailoverAtStep4 from hbase:meta 2023-07-12 15:04:48,042 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testTruncateWithFailoverAtStep4' descriptor. 2023-07-12 15:04:48,042 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testTruncateWithFailoverAtStep4' from region states. 2023-07-12 15:04:48,042 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689174288042"}]},"ts":"9223372036854775807"} 2023-07-12 15:04:48,043 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689174288042"}]},"ts":"9223372036854775807"} 2023-07-12 15:04:48,043 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689174288042"}]},"ts":"9223372036854775807"} 2023-07-12 15:04:48,043 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689174288042"}]},"ts":"9223372036854775807"} 2023-07-12 15:04:48,045 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 4 regions from META 2023-07-12 15:04:48,045 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a317889067b8b7fcd34d1e178c37dcb5, NAME => 'testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5.', STARTKEY => '', ENDKEY => 'a'}, {ENCODED => 717d918e7432e9c5e806f8190b78f43a, NAME => 'testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a.', STARTKEY => 'a', ENDKEY => 'b'}, {ENCODED => 89dfad7a04faa4508cda27c8a2180884, NAME => 'testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884.', STARTKEY => 'b', ENDKEY => 'c'}, {ENCODED => faa8bcd2b476118ae57d0b34a73eb8f0, NAME => 'testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0.', STARTKEY => 'c', ENDKEY => ''}] 2023-07-12 15:04:48,045 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testTruncateWithFailoverAtStep4' as deleted. 2023-07-12 15:04:48,045 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689174288045"}]},"ts":"9223372036854775807"} 2023-07-12 15:04:48,047 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testTruncateWithFailoverAtStep4 state from META 2023-07-12 15:04:48,048 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:48,048 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CREATE_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:48,048 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:48,048 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CREATE_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:48,279 INFO [Listener at localhost/41275] procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:48,279 INFO [Listener at localhost/41275] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:04:48,279 INFO [Listener at localhost/41275] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:48,279 INFO [Listener at localhost/41275] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:48,279 ERROR [Listener at localhost/41275] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:48,279 INFO [Listener at localhost/41275] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:04:48,279 INFO [Listener at localhost/41275] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:48,279 INFO [Listener at localhost/41275] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:48,279 INFO [Listener at localhost/41275] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:48,280 INFO [Listener at localhost/41275] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:48,281 DEBUG [Listener at localhost/41275] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure) stack ids=[0, 1] 2023-07-12 15:04:48,282 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:48,282 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:48,282 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:48,283 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:48,283 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 15:04:48,283 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 15:04:48,283 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:TRUNCATE_TABLE_REMOVE_FROM_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:48,284 DEBUG [Listener at localhost/41275] procedure2.Procedure(966): pid=27, state=RUNNABLE:TRUNCATE_TABLE_REMOVE_FROM_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:48,284 INFO [Listener at localhost/41275] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 3 msec 2023-07-12 15:04:48,284 DEBUG [Listener at localhost/41275] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:48,285 DEBUG [Listener at localhost/41275] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,36649,1689174278222, table=hbase:meta, region=1588230740 2023-07-12 15:04:48,285 INFO [Listener at localhost/41275] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174318285 2023-07-12 15:04:48,285 DEBUG [Listener at localhost/41275] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:48,286 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:TRUNCATE_TABLE_REMOVE_FROM_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:48,288 DEBUG [Listener at localhost/41275] assignment.RegionStateStore(147): Load hbase:meta entry region=2f8aded98651ba93c1387ac5ba7e99cf, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,36649,1689174278222, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222, openSeqNum=2 2023-07-12 15:04:48,289 INFO [Listener at localhost/41275] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:48,289 INFO [Listener at localhost/41275] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174348289 2023-07-12 15:04:48,289 INFO [Listener at localhost/41275] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174408289 2023-07-12 15:04:48,289 INFO [Listener at localhost/41275] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 15:04:48,291 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testTruncateWithFailoverAtStep4' descriptor. 2023-07-12 15:04:48,291 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testTruncateWithFailoverAtStep4' from region states. 2023-07-12 15:04:48,292 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 0 regions from META 2023-07-12 15:04:48,292 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [] 2023-07-12 15:04:48,292 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testTruncateWithFailoverAtStep4' as deleted. 2023-07-12 15:04:48,292 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689174288292"}]},"ts":"9223372036854775807"} 2023-07-12 15:04:48,294 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testTruncateWithFailoverAtStep4 state from META 2023-07-12 15:04:48,295 DEBUG [PEWorker-1] zookeeper.RecoverableZooKeeper(172): Node /hbase/table/testTruncateWithFailoverAtStep4 already deleted, retry=false 2023-07-12 15:04:48,295 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:04:48,301 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40 2023-07-12 15:04:48,301 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:48,301 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f 2023-07-12 15:04:48,301 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a 2023-07-12 15:04:48,302 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40 empty. 2023-07-12 15:04:48,302 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f empty. 2023-07-12 15:04:48,302 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5 empty. 2023-07-12 15:04:48,302 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a empty. 2023-07-12 15:04:48,302 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40 2023-07-12 15:04:48,303 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:48,303 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f 2023-07-12 15:04:48,303 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a 2023-07-12 15:04:48,303 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTruncateWithFailoverAtStep4 regions 2023-07-12 15:04:48,713 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:48,714 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(7675): creating {ENCODED => 01c573fc209a46e97dea2d7a22490a40, NAME => 'testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40.', STARTKEY => '', ENDKEY => 'a'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp 2023-07-12 15:04:48,714 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(7675): creating {ENCODED => 6912f5774859362958fded698274234a, NAME => 'testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a.', STARTKEY => 'b', ENDKEY => 'c'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp 2023-07-12 15:04:48,714 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(7675): creating {ENCODED => abb92fabe2898ac2bafa4af7d1c187a5, NAME => 'testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5.', STARTKEY => 'a', ENDKEY => 'b'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp 2023-07-12 15:04:49,123 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:49,123 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:49,123 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1604): Closing abb92fabe2898ac2bafa4af7d1c187a5, disabling compactions & flushes 2023-07-12 15:04:49,124 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1604): Closing 01c573fc209a46e97dea2d7a22490a40, disabling compactions & flushes 2023-07-12 15:04:49,124 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. 2023-07-12 15:04:49,124 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. 2023-07-12 15:04:49,123 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:49,124 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. after waiting 0 ms 2023-07-12 15:04:49,124 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. 2023-07-12 15:04:49,124 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. 2023-07-12 15:04:49,124 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. 2023-07-12 15:04:49,124 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1604): Closing 6912f5774859362958fded698274234a, disabling compactions & flushes 2023-07-12 15:04:49,124 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. 2023-07-12 15:04:49,124 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. 2023-07-12 15:04:49,124 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. after waiting 0 ms 2023-07-12 15:04:49,124 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. 2023-07-12 15:04:49,124 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. 2023-07-12 15:04:49,124 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. after waiting 0 ms 2023-07-12 15:04:49,124 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. 2023-07-12 15:04:49,124 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. 2023-07-12 15:04:49,124 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1558): Region close journal for abb92fabe2898ac2bafa4af7d1c187a5: 2023-07-12 15:04:49,124 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. 2023-07-12 15:04:49,125 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1558): Region close journal for 6912f5774859362958fded698274234a: 2023-07-12 15:04:49,124 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1558): Region close journal for 01c573fc209a46e97dea2d7a22490a40: 2023-07-12 15:04:49,125 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(7675): creating {ENCODED => 92e418a10c7c6f66245e31ce291aa43f, NAME => 'testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f.', STARTKEY => 'c', ENDKEY => ''}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp 2023-07-12 15:04:49,532 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:49,532 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1604): Closing 92e418a10c7c6f66245e31ce291aa43f, disabling compactions & flushes 2023-07-12 15:04:49,532 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. 2023-07-12 15:04:49,532 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. 2023-07-12 15:04:49,532 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. after waiting 0 ms 2023-07-12 15:04:49,532 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. 2023-07-12 15:04:49,532 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. 2023-07-12 15:04:49,532 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1558): Region close journal for 92e418a10c7c6f66245e31ce291aa43f: 2023-07-12 15:04:49,534 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:49,534 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_ADD_TO_META, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:49,534 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:49,534 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_ADD_TO_META, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:49,539 INFO [Listener at localhost/41275] procedure.TestMasterFailoverWithProcedures(306): Restart 3 exec state: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:49,539 INFO [Listener at localhost/41275] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:04:49,539 INFO [Listener at localhost/41275] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:49,539 INFO [Listener at localhost/41275] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:49,540 ERROR [Listener at localhost/41275] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:49,540 INFO [Listener at localhost/41275] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:04:49,540 INFO [Listener at localhost/41275] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:49,540 INFO [Listener at localhost/41275] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:49,540 INFO [Listener at localhost/41275] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:49,541 INFO [Listener at localhost/41275] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:49,542 DEBUG [Listener at localhost/41275] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure) stack ids=[0, 1, 2] 2023-07-12 15:04:49,542 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:49,542 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:49,542 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:49,542 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:49,543 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 15:04:49,543 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 15:04:49,543 DEBUG [Listener at localhost/41275] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:TRUNCATE_TABLE_CREATE_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:49,543 DEBUG [Listener at localhost/41275] procedure2.Procedure(966): pid=27, state=RUNNABLE:TRUNCATE_TABLE_CREATE_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:49,544 INFO [Listener at localhost/41275] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 15:04:49,544 DEBUG [Listener at localhost/41275] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:49,545 DEBUG [Listener at localhost/41275] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,36649,1689174278222, table=hbase:meta, region=1588230740 2023-07-12 15:04:49,545 INFO [Listener at localhost/41275] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174319545 2023-07-12 15:04:49,546 DEBUG [Listener at localhost/41275] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:49,546 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:TRUNCATE_TABLE_CREATE_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:49,548 DEBUG [Listener at localhost/41275] assignment.RegionStateStore(147): Load hbase:meta entry region=2f8aded98651ba93c1387ac5ba7e99cf, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,36649,1689174278222, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222, openSeqNum=2 2023-07-12 15:04:49,548 INFO [Listener at localhost/41275] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:49,548 INFO [Listener at localhost/41275] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174349548 2023-07-12 15:04:49,548 INFO [Listener at localhost/41275] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174409548 2023-07-12 15:04:49,548 INFO [Listener at localhost/41275] assignment.AssignmentManager(1536): Joined the cluster in 2 msec 2023-07-12 15:04:49,553 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40 2023-07-12 15:04:49,553 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f 2023-07-12 15:04:49,553 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a 2023-07-12 15:04:49,553 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:49,554 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40 empty. 2023-07-12 15:04:49,554 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5 empty. 2023-07-12 15:04:49,555 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f empty. 2023-07-12 15:04:49,555 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a empty. 2023-07-12 15:04:49,555 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40 2023-07-12 15:04:49,555 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:49,556 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f 2023-07-12 15:04:49,556 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a 2023-07-12 15:04:49,556 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTruncateWithFailoverAtStep4 regions 2023-07-12 15:04:49,964 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp/data/default/testTruncateWithFailoverAtStep4/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:49,966 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(7675): creating {ENCODED => 6912f5774859362958fded698274234a, NAME => 'testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a.', STARTKEY => 'b', ENDKEY => 'c'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp 2023-07-12 15:04:49,966 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(7675): creating {ENCODED => abb92fabe2898ac2bafa4af7d1c187a5, NAME => 'testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5.', STARTKEY => 'a', ENDKEY => 'b'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp 2023-07-12 15:04:49,966 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(7675): creating {ENCODED => 01c573fc209a46e97dea2d7a22490a40, NAME => 'testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40.', STARTKEY => '', ENDKEY => 'a'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp 2023-07-12 15:04:49,973 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:49,973 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1604): Closing abb92fabe2898ac2bafa4af7d1c187a5, disabling compactions & flushes 2023-07-12 15:04:49,973 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. 2023-07-12 15:04:49,973 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. 2023-07-12 15:04:49,973 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. after waiting 0 ms 2023-07-12 15:04:49,973 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. 2023-07-12 15:04:49,973 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. 2023-07-12 15:04:49,973 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1558): Region close journal for abb92fabe2898ac2bafa4af7d1c187a5: 2023-07-12 15:04:49,974 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(7675): creating {ENCODED => 92e418a10c7c6f66245e31ce291aa43f, NAME => 'testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f.', STARTKEY => 'c', ENDKEY => ''}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.tmp 2023-07-12 15:04:50,373 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:50,373 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:50,373 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1604): Closing 6912f5774859362958fded698274234a, disabling compactions & flushes 2023-07-12 15:04:50,373 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1604): Closing 01c573fc209a46e97dea2d7a22490a40, disabling compactions & flushes 2023-07-12 15:04:50,373 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. 2023-07-12 15:04:50,373 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. 2023-07-12 15:04:50,373 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. 2023-07-12 15:04:50,373 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. 2023-07-12 15:04:50,373 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. after waiting 0 ms 2023-07-12 15:04:50,373 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. 2023-07-12 15:04:50,373 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. after waiting 0 ms 2023-07-12 15:04:50,373 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. 2023-07-12 15:04:50,373 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. 2023-07-12 15:04:50,373 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. 2023-07-12 15:04:50,374 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1558): Region close journal for 6912f5774859362958fded698274234a: 2023-07-12 15:04:50,374 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1558): Region close journal for 01c573fc209a46e97dea2d7a22490a40: 2023-07-12 15:04:50,379 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:50,379 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1604): Closing 92e418a10c7c6f66245e31ce291aa43f, disabling compactions & flushes 2023-07-12 15:04:50,379 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. 2023-07-12 15:04:50,379 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. 2023-07-12 15:04:50,379 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. after waiting 0 ms 2023-07-12 15:04:50,379 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. 2023-07-12 15:04:50,379 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. 2023-07-12 15:04:50,379 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1558): Region close journal for 92e418a10c7c6f66245e31ce291aa43f: 2023-07-12 15:04:50,381 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:04:50,383 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174290383"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174290383"}]},"ts":"1689174290383"} 2023-07-12 15:04:50,384 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174290383"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174290383"}]},"ts":"1689174290383"} 2023-07-12 15:04:50,384 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174290383"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174290383"}]},"ts":"1689174290383"} 2023-07-12 15:04:50,384 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174290383"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174290383"}]},"ts":"1689174290383"} 2023-07-12 15:04:50,386 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 4 regions to meta. 2023-07-12 15:04:50,386 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:04:50,386 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_ASSIGN_REGIONS, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:50,386 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:50,386 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_ASSIGN_REGIONS, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:04:50,549 INFO [Listener at localhost/41275] procedure.TestMasterFailoverWithProcedures(313): Trigger master failover 2023-07-12 15:04:50,549 INFO [Listener at localhost/41275] hbase.MiniHBaseCluster(607): Aborting jenkins-hbase5.apache.org,37825,1689174278169 2023-07-12 15:04:50,549 ERROR [Listener at localhost/41275] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 15:04:50,549 ERROR [Listener at localhost/41275] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase5.apache.org,37825,1689174278169: Aborting for tests ***** java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:608) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:411) at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.masterFailover(MasterProcedureTestingUtility.java:135) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testRecoveryAndDoubleExecution(TestMasterFailoverWithProcedures.java:314) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testTruncateWithFailoverAtStep(TestMasterFailoverWithProcedures.java:201) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testTruncateWithFailover(TestMasterFailoverWithProcedures.java:175) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:04:50,549 INFO [Listener at localhost/41275] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,37825,1689174278169' ***** 2023-07-12 15:04:50,549 INFO [Listener at localhost/41275] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost/41275 2023-07-12 15:04:50,550 DEBUG [M:0;jenkins-hbase5:37825] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@461d9268, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:04:50,550 INFO [M:0;jenkins-hbase5:37825] regionserver.HRegionServer(1141): aborting server jenkins-hbase5.apache.org,37825,1689174278169 2023-07-12 15:04:50,551 DEBUG [M:0;jenkins-hbase5:37825] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x393e990b to 127.0.0.1:54427 2023-07-12 15:04:50,551 DEBUG [M:0;jenkins-hbase5:37825] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:50,551 INFO [M:0;jenkins-hbase5:37825] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,37825,1689174278169; all regions closed. 2023-07-12 15:04:50,551 DEBUG [M:0;jenkins-hbase5:37825] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:50,551 DEBUG [M:0;jenkins-hbase5:37825] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 15:04:50,551 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 15:04:50,551 DEBUG [M:0;jenkins-hbase5:37825] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 15:04:50,551 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174279577] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174279577,5,FailOnTimeoutGroup] 2023-07-12 15:04:50,552 INFO [M:0;jenkins-hbase5:37825] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 15:04:50,551 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174279578] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174279578,5,FailOnTimeoutGroup] 2023-07-12 15:04:50,552 INFO [M:0;jenkins-hbase5:37825] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 15:04:50,553 INFO [M:0;jenkins-hbase5:37825] hbase.ChoreService(369): Chore service for: master/jenkins-hbase5:0 had [] on shutdown 2023-07-12 15:04:50,553 DEBUG [M:0;jenkins-hbase5:37825] master.HMaster(1512): Stopping service threads 2023-07-12 15:04:50,553 INFO [M:0;jenkins-hbase5:37825] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 15:04:50,553 ERROR [M:0;jenkins-hbase5:37825] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:50,553 INFO [M:0;jenkins-hbase5:37825] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 15:04:50,554 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 15:04:50,554 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:50,554 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): regionserver:36649-0x101c5acc8d60002, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:50,554 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:50,554 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:50,554 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:50,554 DEBUG [M:0;jenkins-hbase5:37825] zookeeper.ZKUtil(398): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 15:04:50,554 WARN [M:0;jenkins-hbase5:37825] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 15:04:50,554 INFO [M:0;jenkins-hbase5:37825] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:50,555 INFO [M:0;jenkins-hbase5:37825] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 15:04:50,557 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/WALs/jenkins-hbase5.apache.org,37825,1689174278169/jenkins-hbase5.apache.org%2C37825%2C1689174278169.1689174279492 not finished, retry = 0 2023-07-12 15:04:50,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:50,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36649-0x101c5acc8d60002, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:50,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:50,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:50,656 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-12 15:04:50,658 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:50,658 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): regionserver:36649-0x101c5acc8d60002, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:50,658 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:50,658 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:50,658 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:50,659 DEBUG [M:0;jenkins-hbase5:37825] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:04:50,659 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:50,659 INFO [M:0;jenkins-hbase5:37825] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:50,659 DEBUG [M:0;jenkins-hbase5:37825] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:50,659 DEBUG [M:0;jenkins-hbase5:37825] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:04:50,659 DEBUG [M:0;jenkins-hbase5:37825] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:50,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:50,659 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase5.apache.org,41959,1689174278198 from backup master directory 2023-07-12 15:04:50,659 ERROR [M:0;jenkins-hbase5:37825] regionserver.HRegion(1825): Memstore data size is 92212 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:50,659 INFO [M:0;jenkins-hbase5:37825] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:50,660 DEBUG [M:0;jenkins-hbase5:37825] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:50,660 INFO [M:0;jenkins-hbase5:37825] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 15:04:50,660 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:04:50,660 INFO [M:0;jenkins-hbase5:37825] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:37825 2023-07-12 15:04:50,661 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:50,661 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase5.apache.org,41959,1689174278198 2023-07-12 15:04:50,661 WARN [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:04:50,661 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:50,661 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase5.apache.org,41959,1689174278198 2023-07-12 15:04:50,663 DEBUG [M:0;jenkins-hbase5:37825] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase5.apache.org,37825,1689174278169 already deleted, retry=false 2023-07-12 15:04:50,674 WARN [RS-EventLoopGroup-14-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase5.apache.org/172.31.10.131:37825 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:37825 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:04:50,675 DEBUG [RS-EventLoopGroup-14-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase5.apache.org/172.31.10.131:37825 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:37825 2023-07-12 15:04:50,684 INFO [master/jenkins-hbase5:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:50,768 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:50,768 INFO [M:0;jenkins-hbase5:37825] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,37825,1689174278169; zookeeper connection closed. 2023-07-12 15:04:50,768 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:37825-0x101c5acc8d60000, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:50,870 DEBUG [RS:0;jenkins-hbase5:36649] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 15:04:50,872 INFO [RS-EventLoopGroup-13-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:38597, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 15:04:50,872 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=41959] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:50,974 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=41959] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:51,076 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=41959] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:51,178 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x738574a4 to 127.0.0.1:54427 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:51,178 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=41959] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:51,182 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7195bc25, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:51,183 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:51,183 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 15:04:51,183 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:51,189 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/WALs/jenkins-hbase5.apache.org,37825,1689174278169 to hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/WALs/jenkins-hbase5.apache.org,37825,1689174278169-dead as it is dead 2023-07-12 15:04:51,189 INFO [master/jenkins-hbase5:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/WALs/jenkins-hbase5.apache.org,37825,1689174278169-dead/jenkins-hbase5.apache.org%2C37825%2C1689174278169.1689174279492 2023-07-12 15:04:51,189 INFO [master/jenkins-hbase5:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/WALs/jenkins-hbase5.apache.org,37825,1689174278169-dead/jenkins-hbase5.apache.org%2C37825%2C1689174278169.1689174279492 after 0ms 2023-07-12 15:04:51,190 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/WALs/jenkins-hbase5.apache.org,37825,1689174278169-dead/jenkins-hbase5.apache.org%2C37825%2C1689174278169.1689174279492 to hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C37825%2C1689174278169.1689174279492 2023-07-12 15:04:51,190 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/WALs/jenkins-hbase5.apache.org,37825,1689174278169-dead 2023-07-12 15:04:51,190 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/WALs/jenkins-hbase5.apache.org,41959,1689174278198 2023-07-12 15:04:51,192 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C41959%2C1689174278198, suffix=, logDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/WALs/jenkins-hbase5.apache.org,41959,1689174278198, archiveDir=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/oldWALs, maxLogs=10 2023-07-12 15:04:51,205 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35761,DS-e518b71d-0c70-439e-b158-b43ca736d25c,DISK] 2023-07-12 15:04:51,207 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/WALs/jenkins-hbase5.apache.org,41959,1689174278198/jenkins-hbase5.apache.org%2C41959%2C1689174278198.1689174291192 2023-07-12 15:04:51,207 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35761,DS-e518b71d-0c70-439e-b158-b43ca736d25c,DISK]] 2023-07-12 15:04:51,207 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:51,207 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:51,208 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:51,208 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:51,209 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:51,209 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 15:04:51,210 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 15:04:51,210 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:51,211 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-12 15:04:51,211 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C37825%2C1689174278169.1689174279492 2023-07-12 15:04:51,218 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 232, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=194, path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C37825%2C1689174278169.1689174279492 2023-07-12 15:04:51,218 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=90.05 KB heapSize=107.48 KB 2023-07-12 15:04:51,282 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=41959] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:51,383 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=41959] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:51,485 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=41959] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:51,587 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=41959] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:04:51,606 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-12 15:04:51,625 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=90.05 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f6b98dc61533403682b5392aea928ec0 2023-07-12 15:04:51,631 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f6b98dc61533403682b5392aea928ec0 as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f6b98dc61533403682b5392aea928ec0 2023-07-12 15:04:51,636 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f6b98dc61533403682b5392aea928ec0, entries=232, sequenceid=194, filesize=95.3 K 2023-07-12 15:04:51,637 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~90.05 KB/92212, heapSize ~107.47 KB/110048, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 419ms, sequenceid=194, compaction requested=false; wal=null 2023-07-12 15:04:51,637 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C37825%2C1689174278169.1689174279492 2023-07-12 15:04:51,640 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:51,642 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/194.seqid, newMaxSeqId=194, maxSeqId=1 2023-07-12 15:04:51,643 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=195; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58727405, jitterRate=-0.12489347159862518}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:04:51,643 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:51,643 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 15:04:51,644 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:51,644 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:51,644 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:51,645 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:51,649 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure) stack ids=[0, 1, 2, 3] 2023-07-12 15:04:51,649 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:51,649 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:04:51,649 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:04:51,650 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:04:51,650 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 15:04:51,650 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 15:04:51,650 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:TRUNCATE_TABLE_ADD_TO_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:51,650 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.Procedure(966): pid=27, state=RUNNABLE:TRUNCATE_TABLE_ADD_TO_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:04:51,651 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-07-12 15:04:51,651 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 15:04:51,651 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:04:51,652 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,36649,1689174278222, table=hbase:meta, region=1588230740 2023-07-12 15:04:51,653 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 1 possibly 'live' servers, and 0 'splitting'. 2023-07-12 15:04:51,653 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:51,664 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 15:04:51,664 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 15:04:51,664 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 15:04:51,664 INFO [master/jenkins-hbase5:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 15:04:51,665 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 15:04:51,665 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 15:04:51,665 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 15:04:51,666 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 15:04:51,666 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase5.apache.org,41959,1689174278198, sessionid=0x101c5acc8d60001, setting cluster-up flag (Was=true) 2023-07-12 15:04:51,668 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 15:04:51,668 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,41959,1689174278198 2023-07-12 15:04:51,671 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 15:04:51,672 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,41959,1689174278198 2023-07-12 15:04:51,673 WARN [master/jenkins-hbase5:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/.hbase-snapshot/.tmp 2023-07-12 15:04:51,673 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:51,673 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:51,674 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:51,674 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:51,674 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase5:0, corePoolSize=10, maxPoolSize=10 2023-07-12 15:04:51,674 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:51,674 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:04:51,674 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:51,674 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174321674 2023-07-12 15:04:51,674 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 15:04:51,675 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:TRUNCATE_TABLE_ADD_TO_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 15:04:51,675 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 15:04:51,675 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 15:04:51,675 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 15:04:51,675 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 15:04:51,675 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 15:04:51,675 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:51,676 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 15:04:51,676 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 15:04:51,676 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 15:04:51,676 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 15:04:51,676 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 15:04:51,676 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174291676,5,FailOnTimeoutGroup] 2023-07-12 15:04:51,677 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174291676,5,FailOnTimeoutGroup] 2023-07-12 15:04:51,677 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:51,677 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 15:04:51,677 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:51,677 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:51,677 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689174291677, completionTime=-1 2023-07-12 15:04:51,677 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 15:04:51,677 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:04:51,678 DEBUG [hconnection-0x654170c0-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:04:51,679 INFO [RS-EventLoopGroup-14-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49214, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:04:51,681 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=2f8aded98651ba93c1387ac5ba7e99cf, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,36649,1689174278222, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222, openSeqNum=2 2023-07-12 15:04:51,682 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=01c573fc209a46e97dea2d7a22490a40, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-12 15:04:51,682 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=abb92fabe2898ac2bafa4af7d1c187a5, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-12 15:04:51,682 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=6912f5774859362958fded698274234a, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-12 15:04:51,682 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=92e418a10c7c6f66245e31ce291aa43f, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-12 15:04:51,682 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:04:51,682 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174351682 2023-07-12 15:04:51,682 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174411682 2023-07-12 15:04:51,682 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-07-12 15:04:51,687 WARN [master/jenkins-hbase5:0:becomeActiveMaster] master.TableStateManager(233): testTruncateWithFailoverAtStep4 has no table state in hbase:meta, assuming ENABLED 2023-07-12 15:04:51,688 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174291687"}]},"ts":"1689174291687"} 2023-07-12 15:04:51,690 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=ENABLED in hbase:meta 2023-07-12 15:04:51,697 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,41959,1689174278198-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:51,697 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,41959,1689174278198-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:51,697 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,41959,1689174278198-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:51,697 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase5:41959, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:51,697 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:51,698 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 15:04:51,700 DEBUG [master/jenkins-hbase5:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 15:04:51,705 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 15:04:51,709 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 15:04:51,709 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 1.048sec 2023-07-12 15:04:51,709 INFO [master/jenkins-hbase5:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 15:04:51,710 INFO [master/jenkins-hbase5:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 15:04:51,710 INFO [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 15:04:51,710 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,41959,1689174278198-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 15:04:51,710 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,41959,1689174278198-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 15:04:51,710 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174291710"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174291710"}]},"ts":"1689174291710"} 2023-07-12 15:04:51,710 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174291710"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174291710"}]},"ts":"1689174291710"} 2023-07-12 15:04:51,711 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174291710"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174291710"}]},"ts":"1689174291710"} 2023-07-12 15:04:51,711 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174291710"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174291710"}]},"ts":"1689174291710"} 2023-07-12 15:04:51,713 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 15:04:51,714 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 4 regions to meta. 2023-07-12 15:04:51,716 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174291716"}]},"ts":"1689174291716"} 2023-07-12 15:04:51,717 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=ENABLING in hbase:meta 2023-07-12 15:04:51,722 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=01c573fc209a46e97dea2d7a22490a40, ASSIGN}, {pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=abb92fabe2898ac2bafa4af7d1c187a5, ASSIGN}, {pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=6912f5774859362958fded698274234a, ASSIGN}, {pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=92e418a10c7c6f66245e31ce291aa43f, ASSIGN}] 2023-07-12 15:04:51,723 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=92e418a10c7c6f66245e31ce291aa43f, ASSIGN 2023-07-12 15:04:51,724 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=92e418a10c7c6f66245e31ce291aa43f, ASSIGN; state=CLOSED, location=jenkins-hbase5.apache.org,36649,1689174278222; forceNewPlan=false, retain=false 2023-07-12 15:04:51,724 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=6912f5774859362958fded698274234a, ASSIGN 2023-07-12 15:04:51,724 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=6912f5774859362958fded698274234a, ASSIGN; state=CLOSED, location=jenkins-hbase5.apache.org,36649,1689174278222; forceNewPlan=false, retain=false 2023-07-12 15:04:51,725 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=abb92fabe2898ac2bafa4af7d1c187a5, ASSIGN 2023-07-12 15:04:51,725 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=abb92fabe2898ac2bafa4af7d1c187a5, ASSIGN; state=CLOSED, location=jenkins-hbase5.apache.org,36649,1689174278222; forceNewPlan=false, retain=false 2023-07-12 15:04:51,726 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=01c573fc209a46e97dea2d7a22490a40, ASSIGN 2023-07-12 15:04:51,726 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=01c573fc209a46e97dea2d7a22490a40, ASSIGN; state=CLOSED, location=jenkins-hbase5.apache.org,36649,1689174278222; forceNewPlan=false, retain=false 2023-07-12 15:04:51,876 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=92e418a10c7c6f66245e31ce291aa43f, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:51,876 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174291876"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174291876"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174291876"}]},"ts":"1689174291876"} 2023-07-12 15:04:51,878 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE; OpenRegionProcedure 92e418a10c7c6f66245e31ce291aa43f, server=jenkins-hbase5.apache.org,36649,1689174278222}] 2023-07-12 15:04:51,881 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=01c573fc209a46e97dea2d7a22490a40, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:51,881 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174291881"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174291881"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174291881"}]},"ts":"1689174291881"} 2023-07-12 15:04:51,884 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=28, state=RUNNABLE; OpenRegionProcedure 01c573fc209a46e97dea2d7a22490a40, server=jenkins-hbase5.apache.org,36649,1689174278222}] 2023-07-12 15:04:51,893 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=abb92fabe2898ac2bafa4af7d1c187a5, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:51,893 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174291893"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174291893"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174291893"}]},"ts":"1689174291893"} 2023-07-12 15:04:51,895 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=29, state=RUNNABLE; OpenRegionProcedure abb92fabe2898ac2bafa4af7d1c187a5, server=jenkins-hbase5.apache.org,36649,1689174278222}] 2023-07-12 15:04:51,897 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=6912f5774859362958fded698274234a, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:51,897 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174291897"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174291897"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174291897"}]},"ts":"1689174291897"} 2023-07-12 15:04:51,898 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=35, ppid=30, state=RUNNABLE; OpenRegionProcedure 6912f5774859362958fded698274234a, server=jenkins-hbase5.apache.org,36649,1689174278222}] 2023-07-12 15:04:52,031 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:52,031 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 15:04:52,032 INFO [RS-EventLoopGroup-14-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49226, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 15:04:52,036 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. 2023-07-12 15:04:52,036 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => abb92fabe2898ac2bafa4af7d1c187a5, NAME => 'testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5.', STARTKEY => 'a', ENDKEY => 'b'} 2023-07-12 15:04:52,036 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:52,036 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:52,036 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:52,036 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:52,037 INFO [StoreOpener-abb92fabe2898ac2bafa4af7d1c187a5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:52,039 DEBUG [StoreOpener-abb92fabe2898ac2bafa4af7d1c187a5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5/f1 2023-07-12 15:04:52,039 DEBUG [StoreOpener-abb92fabe2898ac2bafa4af7d1c187a5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5/f1 2023-07-12 15:04:52,039 INFO [StoreOpener-abb92fabe2898ac2bafa4af7d1c187a5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region abb92fabe2898ac2bafa4af7d1c187a5 columnFamilyName f1 2023-07-12 15:04:52,039 INFO [StoreOpener-abb92fabe2898ac2bafa4af7d1c187a5-1] regionserver.HStore(310): Store=abb92fabe2898ac2bafa4af7d1c187a5/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:52,040 INFO [StoreOpener-abb92fabe2898ac2bafa4af7d1c187a5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:52,041 DEBUG [StoreOpener-abb92fabe2898ac2bafa4af7d1c187a5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5/f2 2023-07-12 15:04:52,041 DEBUG [StoreOpener-abb92fabe2898ac2bafa4af7d1c187a5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5/f2 2023-07-12 15:04:52,041 INFO [StoreOpener-abb92fabe2898ac2bafa4af7d1c187a5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region abb92fabe2898ac2bafa4af7d1c187a5 columnFamilyName f2 2023-07-12 15:04:52,041 INFO [StoreOpener-abb92fabe2898ac2bafa4af7d1c187a5-1] regionserver.HStore(310): Store=abb92fabe2898ac2bafa4af7d1c187a5/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:52,042 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:52,042 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:52,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:04:52,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:52,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:52,047 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened abb92fabe2898ac2bafa4af7d1c187a5; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64750145, jitterRate=-0.03514765202999115}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:04:52,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for abb92fabe2898ac2bafa4af7d1c187a5: 2023-07-12 15:04:52,048 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5., pid=34, masterSystemTime=1689174292031 2023-07-12 15:04:52,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. 2023-07-12 15:04:52,050 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. 2023-07-12 15:04:52,050 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. 2023-07-12 15:04:52,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 92e418a10c7c6f66245e31ce291aa43f, NAME => 'testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f.', STARTKEY => 'c', ENDKEY => ''} 2023-07-12 15:04:52,050 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=abb92fabe2898ac2bafa4af7d1c187a5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:52,051 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174292050"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174292050"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174292050"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174292050"}]},"ts":"1689174292050"} 2023-07-12 15:04:52,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 92e418a10c7c6f66245e31ce291aa43f 2023-07-12 15:04:52,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:52,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 92e418a10c7c6f66245e31ce291aa43f 2023-07-12 15:04:52,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 92e418a10c7c6f66245e31ce291aa43f 2023-07-12 15:04:52,052 INFO [StoreOpener-92e418a10c7c6f66245e31ce291aa43f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 92e418a10c7c6f66245e31ce291aa43f 2023-07-12 15:04:52,053 DEBUG [StoreOpener-92e418a10c7c6f66245e31ce291aa43f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f/f1 2023-07-12 15:04:52,053 DEBUG [StoreOpener-92e418a10c7c6f66245e31ce291aa43f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f/f1 2023-07-12 15:04:52,053 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=29 2023-07-12 15:04:52,053 INFO [StoreOpener-92e418a10c7c6f66245e31ce291aa43f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92e418a10c7c6f66245e31ce291aa43f columnFamilyName f1 2023-07-12 15:04:52,053 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=29, state=SUCCESS; OpenRegionProcedure abb92fabe2898ac2bafa4af7d1c187a5, server=jenkins-hbase5.apache.org,36649,1689174278222 in 157 msec 2023-07-12 15:04:52,054 INFO [StoreOpener-92e418a10c7c6f66245e31ce291aa43f-1] regionserver.HStore(310): Store=92e418a10c7c6f66245e31ce291aa43f/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:52,054 INFO [StoreOpener-92e418a10c7c6f66245e31ce291aa43f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 92e418a10c7c6f66245e31ce291aa43f 2023-07-12 15:04:52,055 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=abb92fabe2898ac2bafa4af7d1c187a5, ASSIGN in 332 msec 2023-07-12 15:04:52,055 DEBUG [StoreOpener-92e418a10c7c6f66245e31ce291aa43f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f/f2 2023-07-12 15:04:52,055 DEBUG [StoreOpener-92e418a10c7c6f66245e31ce291aa43f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f/f2 2023-07-12 15:04:52,055 INFO [StoreOpener-92e418a10c7c6f66245e31ce291aa43f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92e418a10c7c6f66245e31ce291aa43f columnFamilyName f2 2023-07-12 15:04:52,056 INFO [StoreOpener-92e418a10c7c6f66245e31ce291aa43f-1] regionserver.HStore(310): Store=92e418a10c7c6f66245e31ce291aa43f/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:52,057 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f 2023-07-12 15:04:52,057 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f 2023-07-12 15:04:52,058 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:04:52,059 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 92e418a10c7c6f66245e31ce291aa43f 2023-07-12 15:04:52,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:52,061 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 92e418a10c7c6f66245e31ce291aa43f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59644492, jitterRate=-0.11122781038284302}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:04:52,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 92e418a10c7c6f66245e31ce291aa43f: 2023-07-12 15:04:52,062 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f., pid=32, masterSystemTime=1689174292031 2023-07-12 15:04:52,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. 2023-07-12 15:04:52,064 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. 2023-07-12 15:04:52,064 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. 2023-07-12 15:04:52,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 01c573fc209a46e97dea2d7a22490a40, NAME => 'testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40.', STARTKEY => '', ENDKEY => 'a'} 2023-07-12 15:04:52,064 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=92e418a10c7c6f66245e31ce291aa43f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:52,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 01c573fc209a46e97dea2d7a22490a40 2023-07-12 15:04:52,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:52,064 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174292064"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174292064"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174292064"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174292064"}]},"ts":"1689174292064"} 2023-07-12 15:04:52,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 01c573fc209a46e97dea2d7a22490a40 2023-07-12 15:04:52,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 01c573fc209a46e97dea2d7a22490a40 2023-07-12 15:04:52,067 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-07-12 15:04:52,067 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; OpenRegionProcedure 92e418a10c7c6f66245e31ce291aa43f, server=jenkins-hbase5.apache.org,36649,1689174278222 in 188 msec 2023-07-12 15:04:52,067 INFO [StoreOpener-01c573fc209a46e97dea2d7a22490a40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 01c573fc209a46e97dea2d7a22490a40 2023-07-12 15:04:52,068 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=92e418a10c7c6f66245e31ce291aa43f, ASSIGN in 345 msec 2023-07-12 15:04:52,068 DEBUG [StoreOpener-01c573fc209a46e97dea2d7a22490a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40/f1 2023-07-12 15:04:52,068 DEBUG [StoreOpener-01c573fc209a46e97dea2d7a22490a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40/f1 2023-07-12 15:04:52,069 INFO [StoreOpener-01c573fc209a46e97dea2d7a22490a40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 01c573fc209a46e97dea2d7a22490a40 columnFamilyName f1 2023-07-12 15:04:52,069 INFO [StoreOpener-01c573fc209a46e97dea2d7a22490a40-1] regionserver.HStore(310): Store=01c573fc209a46e97dea2d7a22490a40/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:52,069 INFO [StoreOpener-01c573fc209a46e97dea2d7a22490a40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 01c573fc209a46e97dea2d7a22490a40 2023-07-12 15:04:52,070 DEBUG [StoreOpener-01c573fc209a46e97dea2d7a22490a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40/f2 2023-07-12 15:04:52,070 DEBUG [StoreOpener-01c573fc209a46e97dea2d7a22490a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40/f2 2023-07-12 15:04:52,071 INFO [StoreOpener-01c573fc209a46e97dea2d7a22490a40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 01c573fc209a46e97dea2d7a22490a40 columnFamilyName f2 2023-07-12 15:04:52,071 INFO [StoreOpener-01c573fc209a46e97dea2d7a22490a40-1] regionserver.HStore(310): Store=01c573fc209a46e97dea2d7a22490a40/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:52,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40 2023-07-12 15:04:52,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40 2023-07-12 15:04:52,073 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:04:52,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 01c573fc209a46e97dea2d7a22490a40 2023-07-12 15:04:52,076 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:52,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 01c573fc209a46e97dea2d7a22490a40; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61563800, jitterRate=-0.08262789249420166}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:04:52,076 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 01c573fc209a46e97dea2d7a22490a40: 2023-07-12 15:04:52,077 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40., pid=33, masterSystemTime=1689174292031 2023-07-12 15:04:52,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. 2023-07-12 15:04:52,078 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. 2023-07-12 15:04:52,078 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. 2023-07-12 15:04:52,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6912f5774859362958fded698274234a, NAME => 'testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a.', STARTKEY => 'b', ENDKEY => 'c'} 2023-07-12 15:04:52,078 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=01c573fc209a46e97dea2d7a22490a40, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:52,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 6912f5774859362958fded698274234a 2023-07-12 15:04:52,079 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689174292078"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174292078"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174292078"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174292078"}]},"ts":"1689174292078"} 2023-07-12 15:04:52,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:52,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 6912f5774859362958fded698274234a 2023-07-12 15:04:52,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 6912f5774859362958fded698274234a 2023-07-12 15:04:52,080 INFO [StoreOpener-6912f5774859362958fded698274234a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 6912f5774859362958fded698274234a 2023-07-12 15:04:52,081 DEBUG [StoreOpener-6912f5774859362958fded698274234a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a/f1 2023-07-12 15:04:52,081 DEBUG [StoreOpener-6912f5774859362958fded698274234a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a/f1 2023-07-12 15:04:52,082 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=28 2023-07-12 15:04:52,082 INFO [StoreOpener-6912f5774859362958fded698274234a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6912f5774859362958fded698274234a columnFamilyName f1 2023-07-12 15:04:52,082 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=28, state=SUCCESS; OpenRegionProcedure 01c573fc209a46e97dea2d7a22490a40, server=jenkins-hbase5.apache.org,36649,1689174278222 in 197 msec 2023-07-12 15:04:52,082 INFO [StoreOpener-6912f5774859362958fded698274234a-1] regionserver.HStore(310): Store=6912f5774859362958fded698274234a/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:52,082 INFO [StoreOpener-6912f5774859362958fded698274234a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 6912f5774859362958fded698274234a 2023-07-12 15:04:52,083 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=01c573fc209a46e97dea2d7a22490a40, ASSIGN in 360 msec 2023-07-12 15:04:52,083 DEBUG [StoreOpener-6912f5774859362958fded698274234a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a/f2 2023-07-12 15:04:52,084 DEBUG [StoreOpener-6912f5774859362958fded698274234a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a/f2 2023-07-12 15:04:52,084 INFO [StoreOpener-6912f5774859362958fded698274234a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6912f5774859362958fded698274234a columnFamilyName f2 2023-07-12 15:04:52,084 INFO [StoreOpener-6912f5774859362958fded698274234a-1] regionserver.HStore(310): Store=6912f5774859362958fded698274234a/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:52,085 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a 2023-07-12 15:04:52,085 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a 2023-07-12 15:04:52,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:04:52,088 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 6912f5774859362958fded698274234a 2023-07-12 15:04:52,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:52,090 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 6912f5774859362958fded698274234a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58843563, jitterRate=-0.12316258251667023}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:04:52,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 6912f5774859362958fded698274234a: 2023-07-12 15:04:52,090 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a., pid=35, masterSystemTime=1689174292031 2023-07-12 15:04:52,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. 2023-07-12 15:04:52,092 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. 2023-07-12 15:04:52,092 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=6912f5774859362958fded698274234a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:52,092 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689174292092"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174292092"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174292092"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174292092"}]},"ts":"1689174292092"} 2023-07-12 15:04:52,094 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=35, resume processing ppid=30 2023-07-12 15:04:52,095 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, ppid=30, state=SUCCESS; OpenRegionProcedure 6912f5774859362958fded698274234a, server=jenkins-hbase5.apache.org,36649,1689174278222 in 195 msec 2023-07-12 15:04:52,096 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=27 2023-07-12 15:04:52,096 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=6912f5774859362958fded698274234a, ASSIGN in 373 msec 2023-07-12 15:04:52,096 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174292096"}]},"ts":"1689174292096"} 2023-07-12 15:04:52,097 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=ENABLED in hbase:meta 2023-07-12 15:04:52,099 DEBUG [PEWorker-1] procedure.TruncateTableProcedure(145): truncate 'testTruncateWithFailoverAtStep4' completed 2023-07-12 15:04:52,099 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) in 4.6010 sec 2023-07-12 15:04:52,304 WARN [Listener at localhost/41275] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-12 15:04:52,305 DEBUG [Listener at localhost/41275] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testTruncateWithFailoverAtStep4 get assigned. Timeout = 60000ms 2023-07-12 15:04:52,306 INFO [Listener at localhost/41275] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:04:52,309 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(3484): All regions for table testTruncateWithFailoverAtStep4 assigned to meta. Checking AM states. 2023-07-12 15:04:52,310 INFO [Listener at localhost/41275] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:04:52,310 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(3504): All regions for table testTruncateWithFailoverAtStep4 assigned. 2023-07-12 15:04:52,313 DEBUG [Listener at localhost/41275] util.CommonFSUtils(682): File system contents for path hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4 2023-07-12 15:04:52,314 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |-.tabledesc/ 2023-07-12 15:04:52,314 DEBUG [Listener at localhost/41275] util.CommonFSUtils(702): |----.tableinfo.0000000001 2023-07-12 15:04:52,315 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |-.tmp/ 2023-07-12 15:04:52,315 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |-01c573fc209a46e97dea2d7a22490a40/ 2023-07-12 15:04:52,315 DEBUG [Listener at localhost/41275] util.CommonFSUtils(702): |----.regioninfo 2023-07-12 15:04:52,315 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |----f1/ 2023-07-12 15:04:52,316 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |----f2/ 2023-07-12 15:04:52,316 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-12 15:04:52,316 DEBUG [Listener at localhost/41275] util.CommonFSUtils(702): |-------1.seqid 2023-07-12 15:04:52,317 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |-6912f5774859362958fded698274234a/ 2023-07-12 15:04:52,317 DEBUG [Listener at localhost/41275] util.CommonFSUtils(702): |----.regioninfo 2023-07-12 15:04:52,317 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |----f1/ 2023-07-12 15:04:52,317 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |----f2/ 2023-07-12 15:04:52,318 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-12 15:04:52,318 DEBUG [Listener at localhost/41275] util.CommonFSUtils(702): |-------1.seqid 2023-07-12 15:04:52,318 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |-92e418a10c7c6f66245e31ce291aa43f/ 2023-07-12 15:04:52,318 DEBUG [Listener at localhost/41275] util.CommonFSUtils(702): |----.regioninfo 2023-07-12 15:04:52,318 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |----f1/ 2023-07-12 15:04:52,319 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |----f2/ 2023-07-12 15:04:52,319 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-12 15:04:52,319 DEBUG [Listener at localhost/41275] util.CommonFSUtils(702): |-------1.seqid 2023-07-12 15:04:52,319 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |-abb92fabe2898ac2bafa4af7d1c187a5/ 2023-07-12 15:04:52,320 DEBUG [Listener at localhost/41275] util.CommonFSUtils(702): |----.regioninfo 2023-07-12 15:04:52,320 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |----f1/ 2023-07-12 15:04:52,320 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |----f2/ 2023-07-12 15:04:52,320 DEBUG [Listener at localhost/41275] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-12 15:04:52,321 DEBUG [Listener at localhost/41275] util.CommonFSUtils(702): |-------1.seqid 2023-07-12 15:04:52,327 DEBUG [Listener at localhost/41275] procedure.MasterProcedureTestingUtility(212): Table directory layout is as expected. 2023-07-12 15:04:52,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36649] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Scan size: 154 connection: 172.31.10.131:46122 deadline: 1689174352333, exception=org.apache.hadoop.hbase.NotServingRegionException: testTruncateWithFailoverAtStep4,,1689174282050.a317889067b8b7fcd34d1e178c37dcb5. is not online on jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:52,443 DEBUG [Listener at localhost/41275] client.ClientScanner(240): Advancing internal scanner to startKey at 'a', inclusive 2023-07-12 15:04:52,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36649] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Scan size: 158 connection: 172.31.10.131:46122 deadline: 1689174352444, exception=org.apache.hadoop.hbase.NotServingRegionException: testTruncateWithFailoverAtStep4,a,1689174282050.717d918e7432e9c5e806f8190b78f43a. is not online on jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:52,549 DEBUG [Listener at localhost/41275] client.ClientScanner(240): Advancing internal scanner to startKey at 'b', inclusive 2023-07-12 15:04:52,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36649] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Scan size: 158 connection: 172.31.10.131:46122 deadline: 1689174352550, exception=org.apache.hadoop.hbase.NotServingRegionException: testTruncateWithFailoverAtStep4,b,1689174282050.89dfad7a04faa4508cda27c8a2180884. is not online on jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:52,655 DEBUG [Listener at localhost/41275] client.ClientScanner(240): Advancing internal scanner to startKey at 'c', inclusive 2023-07-12 15:04:52,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36649] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Scan size: 158 connection: 172.31.10.131:46122 deadline: 1689174352656, exception=org.apache.hadoop.hbase.NotServingRegionException: testTruncateWithFailoverAtStep4,c,1689174282050.faa8bcd2b476118ae57d0b34a73eb8f0. is not online on jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:52,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36649] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. with WAL disabled. Data may be lost in the event of a crash. 2023-07-12 15:04:52,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36649] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. with WAL disabled. Data may be lost in the event of a crash. 2023-07-12 15:04:52,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36649] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. with WAL disabled. Data may be lost in the event of a crash. 2023-07-12 15:04:52,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36649] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. with WAL disabled. Data may be lost in the event of a crash. 2023-07-12 15:04:52,770 DEBUG [Listener at localhost/41275] client.ClientScanner(240): Advancing internal scanner to startKey at 'a', inclusive 2023-07-12 15:04:52,771 DEBUG [Listener at localhost/41275] client.ClientScanner(240): Advancing internal scanner to startKey at 'b', inclusive 2023-07-12 15:04:52,772 DEBUG [Listener at localhost/41275] client.ClientScanner(240): Advancing internal scanner to startKey at 'c', inclusive 2023-07-12 15:04:52,773 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 15:04:52,774 INFO [Listener at localhost/41275] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 15:04:52,774 DEBUG [Listener at localhost/41275] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c281a1e to 127.0.0.1:54427 2023-07-12 15:04:52,774 DEBUG [Listener at localhost/41275] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:52,774 DEBUG [Listener at localhost/41275] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 15:04:52,774 DEBUG [Listener at localhost/41275] util.JVMClusterUtil(257): Found active master hash=776449671, stopped=false 2023-07-12 15:04:52,774 INFO [Listener at localhost/41275] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase5.apache.org,41959,1689174278198 2023-07-12 15:04:52,777 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:52,777 INFO [Listener at localhost/41275] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:04:52,777 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:52,777 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): regionserver:36649-0x101c5acc8d60002, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:52,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:52,778 DEBUG [Listener at localhost/41275] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x738574a4 to 127.0.0.1:54427 2023-07-12 15:04:52,778 DEBUG [Listener at localhost/41275] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:52,778 INFO [Listener at localhost/41275] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,36649,1689174278222' ***** 2023-07-12 15:04:52,779 INFO [Listener at localhost/41275] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 15:04:52,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36649-0x101c5acc8d60002, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:52,779 INFO [RS:0;jenkins-hbase5:36649] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 15:04:52,779 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 15:04:52,779 INFO [RS:0;jenkins-hbase5:36649] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 15:04:52,780 INFO [RS:0;jenkins-hbase5:36649] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 15:04:52,780 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(3305): Received CLOSE for 01c573fc209a46e97dea2d7a22490a40 2023-07-12 15:04:52,780 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(3305): Received CLOSE for 92e418a10c7c6f66245e31ce291aa43f 2023-07-12 15:04:52,780 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(3305): Received CLOSE for 2f8aded98651ba93c1387ac5ba7e99cf 2023-07-12 15:04:52,780 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(3305): Received CLOSE for abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:52,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 01c573fc209a46e97dea2d7a22490a40, disabling compactions & flushes 2023-07-12 15:04:52,780 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(3305): Received CLOSE for 6912f5774859362958fded698274234a 2023-07-12 15:04:52,780 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. 2023-07-12 15:04:52,780 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:52,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. 2023-07-12 15:04:52,780 DEBUG [RS:0;jenkins-hbase5:36649] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x780f8f53 to 127.0.0.1:54427 2023-07-12 15:04:52,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. after waiting 0 ms 2023-07-12 15:04:52,780 DEBUG [RS:0;jenkins-hbase5:36649] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:52,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. 2023-07-12 15:04:52,781 INFO [RS:0;jenkins-hbase5:36649] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 15:04:52,781 INFO [RS:0;jenkins-hbase5:36649] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 15:04:52,781 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 01c573fc209a46e97dea2d7a22490a40 2/2 column families, dataSize=3.79 KB heapSize=8.66 KB 2023-07-12 15:04:52,781 INFO [RS:0;jenkins-hbase5:36649] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 15:04:52,781 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 15:04:52,781 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1474): Waiting on 6 regions to close 2023-07-12 15:04:52,781 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1478): Online Regions={01c573fc209a46e97dea2d7a22490a40=testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40., 92e418a10c7c6f66245e31ce291aa43f=testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f., 2f8aded98651ba93c1387ac5ba7e99cf=hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf., abb92fabe2898ac2bafa4af7d1c187a5=testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5., 1588230740=hbase:meta,,1.1588230740, 6912f5774859362958fded698274234a=testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a.} 2023-07-12 15:04:52,781 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 15:04:52,781 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 01c573fc209a46e97dea2d7a22490a40, 1588230740, 2f8aded98651ba93c1387ac5ba7e99cf, 6912f5774859362958fded698274234a, 92e418a10c7c6f66245e31ce291aa43f, abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:52,781 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 15:04:52,781 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 15:04:52,781 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 15:04:52,781 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 15:04:52,781 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=18.80 KB heapSize=30.18 KB 2023-07-12 15:04:52,792 INFO [regionserver/jenkins-hbase5:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 15:04:52,981 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 01c573fc209a46e97dea2d7a22490a40, 1588230740, 2f8aded98651ba93c1387ac5ba7e99cf, 6912f5774859362958fded698274234a, 92e418a10c7c6f66245e31ce291aa43f, abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:53,181 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 01c573fc209a46e97dea2d7a22490a40, 1588230740, 2f8aded98651ba93c1387ac5ba7e99cf, 6912f5774859362958fded698274234a, 92e418a10c7c6f66245e31ce291aa43f, abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:53,190 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.90 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40/.tmp/f1/ec09150c7fe44392a7fdfd5b7794b745 2023-07-12 15:04:53,191 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.31 KB at sequenceid=47 (bloomFilter=false), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/.tmp/info/bbf88cbf97144dafafe5911b5a2a1c4a 2023-07-12 15:04:53,197 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bbf88cbf97144dafafe5911b5a2a1c4a 2023-07-12 15:04:53,382 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 01c573fc209a46e97dea2d7a22490a40, 1588230740, 2f8aded98651ba93c1387ac5ba7e99cf, 6912f5774859362958fded698274234a, 92e418a10c7c6f66245e31ce291aa43f, abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:53,582 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 01c573fc209a46e97dea2d7a22490a40, 1588230740, 2f8aded98651ba93c1387ac5ba7e99cf, 6912f5774859362958fded698274234a, 92e418a10c7c6f66245e31ce291aa43f, abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:53,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.90 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40/.tmp/f2/c877a347d2514a499edc7c50d92d5199 2023-07-12 15:04:53,607 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=447 B at sequenceid=47 (bloomFilter=false), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/.tmp/rep_barrier/51828fed67694769b14ef827e4c310fb 2023-07-12 15:04:53,610 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40/.tmp/f1/ec09150c7fe44392a7fdfd5b7794b745 as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40/f1/ec09150c7fe44392a7fdfd5b7794b745 2023-07-12 15:04:53,612 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 51828fed67694769b14ef827e4c310fb 2023-07-12 15:04:53,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40/f1/ec09150c7fe44392a7fdfd5b7794b745, entries=29, sequenceid=5, filesize=6.8 K 2023-07-12 15:04:53,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40/.tmp/f2/c877a347d2514a499edc7c50d92d5199 as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40/f2/c877a347d2514a499edc7c50d92d5199 2023-07-12 15:04:53,622 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40/f2/c877a347d2514a499edc7c50d92d5199, entries=29, sequenceid=5, filesize=6.8 K 2023-07-12 15:04:53,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.79 KB/3886, heapSize ~8.63 KB/8832, currentSize=0 B/0 for 01c573fc209a46e97dea2d7a22490a40 in 842ms, sequenceid=5, compaction requested=false 2023-07-12 15:04:53,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testTruncateWithFailoverAtStep4' 2023-07-12 15:04:53,636 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/01c573fc209a46e97dea2d7a22490a40/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 15:04:53,636 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. 2023-07-12 15:04:53,636 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 01c573fc209a46e97dea2d7a22490a40: 2023-07-12 15:04:53,636 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testTruncateWithFailoverAtStep4,,1689174288036.01c573fc209a46e97dea2d7a22490a40. 2023-07-12 15:04:53,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 92e418a10c7c6f66245e31ce291aa43f, disabling compactions & flushes 2023-07-12 15:04:53,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. 2023-07-12 15:04:53,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. 2023-07-12 15:04:53,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. after waiting 0 ms 2023-07-12 15:04:53,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. 2023-07-12 15:04:53,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 92e418a10c7c6f66245e31ce291aa43f 2/2 column families, dataSize=1.70 KB heapSize=4.16 KB 2023-07-12 15:04:53,720 INFO [regionserver/jenkins-hbase5:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-12 15:04:53,721 INFO [regionserver/jenkins-hbase5:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-12 15:04:53,782 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1474): Waiting on 5 regions to close 2023-07-12 15:04:53,782 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1478): Online Regions={92e418a10c7c6f66245e31ce291aa43f=testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f., 2f8aded98651ba93c1387ac5ba7e99cf=hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf., abb92fabe2898ac2bafa4af7d1c187a5=testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5., 1588230740=hbase:meta,,1.1588230740, 6912f5774859362958fded698274234a=testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a.} 2023-07-12 15:04:53,783 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 1588230740, 2f8aded98651ba93c1387ac5ba7e99cf, 6912f5774859362958fded698274234a, 92e418a10c7c6f66245e31ce291aa43f, abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:53,983 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 1588230740, 2f8aded98651ba93c1387ac5ba7e99cf, 6912f5774859362958fded698274234a, 92e418a10c7c6f66245e31ce291aa43f, abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:54,023 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.05 KB at sequenceid=47 (bloomFilter=false), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/.tmp/table/a6d98d14397c49259b0128e6be56801e 2023-07-12 15:04:54,028 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a6d98d14397c49259b0128e6be56801e 2023-07-12 15:04:54,029 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/.tmp/info/bbf88cbf97144dafafe5911b5a2a1c4a as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/info/bbf88cbf97144dafafe5911b5a2a1c4a 2023-07-12 15:04:54,034 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bbf88cbf97144dafafe5911b5a2a1c4a 2023-07-12 15:04:54,034 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/info/bbf88cbf97144dafafe5911b5a2a1c4a, entries=58, sequenceid=47, filesize=12.6 K 2023-07-12 15:04:54,035 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/.tmp/rep_barrier/51828fed67694769b14ef827e4c310fb as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/rep_barrier/51828fed67694769b14ef827e4c310fb 2023-07-12 15:04:54,040 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 51828fed67694769b14ef827e4c310fb 2023-07-12 15:04:54,040 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/rep_barrier/51828fed67694769b14ef827e4c310fb, entries=4, sequenceid=47, filesize=5.4 K 2023-07-12 15:04:54,041 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/.tmp/table/a6d98d14397c49259b0128e6be56801e as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/table/a6d98d14397c49259b0128e6be56801e 2023-07-12 15:04:54,045 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=869 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f/.tmp/f1/5ab8e674ab3e43c2a5b6736c1bc12d7e 2023-07-12 15:04:54,046 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a6d98d14397c49259b0128e6be56801e 2023-07-12 15:04:54,046 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/table/a6d98d14397c49259b0128e6be56801e, entries=11, sequenceid=47, filesize=5.7 K 2023-07-12 15:04:54,047 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~18.80 KB/19256, heapSize ~30.13 KB/30856, currentSize=0 B/0 for 1588230740 in 1266ms, sequenceid=47, compaction requested=false 2023-07-12 15:04:54,054 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/meta/1588230740/recovered.edits/50.seqid, newMaxSeqId=50, maxSeqId=1 2023-07-12 15:04:54,055 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 15:04:54,055 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 15:04:54,055 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 15:04:54,055 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 15:04:54,183 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 2f8aded98651ba93c1387ac5ba7e99cf, 6912f5774859362958fded698274234a, 92e418a10c7c6f66245e31ce291aa43f, abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:54,383 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 2f8aded98651ba93c1387ac5ba7e99cf, 6912f5774859362958fded698274234a, 92e418a10c7c6f66245e31ce291aa43f, abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:54,460 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=869 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f/.tmp/f2/efde00a0e5d041af9f287baed506abfc 2023-07-12 15:04:54,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f/.tmp/f1/5ab8e674ab3e43c2a5b6736c1bc12d7e as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f/f1/5ab8e674ab3e43c2a5b6736c1bc12d7e 2023-07-12 15:04:54,471 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f/f1/5ab8e674ab3e43c2a5b6736c1bc12d7e, entries=13, sequenceid=5, filesize=5.7 K 2023-07-12 15:04:54,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f/.tmp/f2/efde00a0e5d041af9f287baed506abfc as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f/f2/efde00a0e5d041af9f287baed506abfc 2023-07-12 15:04:54,477 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f/f2/efde00a0e5d041af9f287baed506abfc, entries=13, sequenceid=5, filesize=5.7 K 2023-07-12 15:04:54,478 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.70 KB/1738, heapSize ~4.13 KB/4224, currentSize=0 B/0 for 92e418a10c7c6f66245e31ce291aa43f in 841ms, sequenceid=5, compaction requested=false 2023-07-12 15:04:54,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/92e418a10c7c6f66245e31ce291aa43f/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 15:04:54,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. 2023-07-12 15:04:54,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 92e418a10c7c6f66245e31ce291aa43f: 2023-07-12 15:04:54,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testTruncateWithFailoverAtStep4,c,1689174288036.92e418a10c7c6f66245e31ce291aa43f. 2023-07-12 15:04:54,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 2f8aded98651ba93c1387ac5ba7e99cf, disabling compactions & flushes 2023-07-12 15:04:54,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf. 2023-07-12 15:04:54,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf. 2023-07-12 15:04:54,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf. after waiting 0 ms 2023-07-12 15:04:54,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf. 2023-07-12 15:04:54,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 2f8aded98651ba93c1387ac5ba7e99cf 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-12 15:04:54,583 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 2f8aded98651ba93c1387ac5ba7e99cf, 6912f5774859362958fded698274234a, abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:54,784 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-12 15:04:54,784 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1478): Online Regions={2f8aded98651ba93c1387ac5ba7e99cf=hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf., abb92fabe2898ac2bafa4af7d1c187a5=testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5., 6912f5774859362958fded698274234a=testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a.} 2023-07-12 15:04:54,784 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 2f8aded98651ba93c1387ac5ba7e99cf, 6912f5774859362958fded698274234a, abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:54,892 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/namespace/2f8aded98651ba93c1387ac5ba7e99cf/.tmp/info/d7d86fa4295141c480a06a9795a74a07 2023-07-12 15:04:54,898 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/namespace/2f8aded98651ba93c1387ac5ba7e99cf/.tmp/info/d7d86fa4295141c480a06a9795a74a07 as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/namespace/2f8aded98651ba93c1387ac5ba7e99cf/info/d7d86fa4295141c480a06a9795a74a07 2023-07-12 15:04:54,903 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/namespace/2f8aded98651ba93c1387ac5ba7e99cf/info/d7d86fa4295141c480a06a9795a74a07, entries=2, sequenceid=6, filesize=4.8 K 2023-07-12 15:04:54,904 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 2f8aded98651ba93c1387ac5ba7e99cf in 420ms, sequenceid=6, compaction requested=false 2023-07-12 15:04:54,909 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/hbase/namespace/2f8aded98651ba93c1387ac5ba7e99cf/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-12 15:04:54,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf. 2023-07-12 15:04:54,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 2f8aded98651ba93c1387ac5ba7e99cf: 2023-07-12 15:04:54,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689174280790.2f8aded98651ba93c1387ac5ba7e99cf. 2023-07-12 15:04:54,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing abb92fabe2898ac2bafa4af7d1c187a5, disabling compactions & flushes 2023-07-12 15:04:54,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. 2023-07-12 15:04:54,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. 2023-07-12 15:04:54,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. after waiting 0 ms 2023-07-12 15:04:54,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. 2023-07-12 15:04:54,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing abb92fabe2898ac2bafa4af7d1c187a5 2/2 column families, dataSize=532 B heapSize=1.63 KB 2023-07-12 15:04:54,984 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 6912f5774859362958fded698274234a, abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:55,184 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 6912f5774859362958fded698274234a, abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:55,317 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=266 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5/.tmp/f1/1d7913b0297a44c5b0f24d977a4a0ac5 2023-07-12 15:04:55,385 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 6912f5774859362958fded698274234a, abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:55,585 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 6912f5774859362958fded698274234a, abb92fabe2898ac2bafa4af7d1c187a5 2023-07-12 15:04:55,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=266 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5/.tmp/f2/ea20aeffb1fc4efca3bdcabf6151a86f 2023-07-12 15:04:55,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5/.tmp/f1/1d7913b0297a44c5b0f24d977a4a0ac5 as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5/f1/1d7913b0297a44c5b0f24d977a4a0ac5 2023-07-12 15:04:55,740 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5/f1/1d7913b0297a44c5b0f24d977a4a0ac5, entries=4, sequenceid=5, filesize=5.1 K 2023-07-12 15:04:55,741 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5/.tmp/f2/ea20aeffb1fc4efca3bdcabf6151a86f as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5/f2/ea20aeffb1fc4efca3bdcabf6151a86f 2023-07-12 15:04:55,745 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5/f2/ea20aeffb1fc4efca3bdcabf6151a86f, entries=4, sequenceid=5, filesize=5.1 K 2023-07-12 15:04:55,746 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~532 B/532, heapSize ~1.59 KB/1632, currentSize=0 B/0 for abb92fabe2898ac2bafa4af7d1c187a5 in 836ms, sequenceid=5, compaction requested=false 2023-07-12 15:04:55,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/abb92fabe2898ac2bafa4af7d1c187a5/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 15:04:55,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. 2023-07-12 15:04:55,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for abb92fabe2898ac2bafa4af7d1c187a5: 2023-07-12 15:04:55,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testTruncateWithFailoverAtStep4,a,1689174288036.abb92fabe2898ac2bafa4af7d1c187a5. 2023-07-12 15:04:55,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 6912f5774859362958fded698274234a, disabling compactions & flushes 2023-07-12 15:04:55,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. 2023-07-12 15:04:55,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. 2023-07-12 15:04:55,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. after waiting 0 ms 2023-07-12 15:04:55,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. 2023-07-12 15:04:55,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 6912f5774859362958fded698274234a 2/2 column families, dataSize=532 B heapSize=1.63 KB 2023-07-12 15:04:55,785 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-12 15:04:55,785 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1478): Online Regions={6912f5774859362958fded698274234a=testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a.} 2023-07-12 15:04:55,785 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 6912f5774859362958fded698274234a 2023-07-12 15:04:55,985 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 6912f5774859362958fded698274234a 2023-07-12 15:04:56,159 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=266 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a/.tmp/f1/dd58b7d42d594634b9fd2ff22ebca8a3 2023-07-12 15:04:56,186 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 6912f5774859362958fded698274234a 2023-07-12 15:04:56,386 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 6912f5774859362958fded698274234a 2023-07-12 15:04:56,571 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=266 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a/.tmp/f2/a375ecc870d447d9a3fdefd4b0f60df0 2023-07-12 15:04:56,576 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a/.tmp/f1/dd58b7d42d594634b9fd2ff22ebca8a3 as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a/f1/dd58b7d42d594634b9fd2ff22ebca8a3 2023-07-12 15:04:56,581 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a/f1/dd58b7d42d594634b9fd2ff22ebca8a3, entries=4, sequenceid=5, filesize=5.1 K 2023-07-12 15:04:56,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a/.tmp/f2/a375ecc870d447d9a3fdefd4b0f60df0 as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a/f2/a375ecc870d447d9a3fdefd4b0f60df0 2023-07-12 15:04:56,586 DEBUG [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1504): Waiting on 6912f5774859362958fded698274234a 2023-07-12 15:04:56,587 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a/f2/a375ecc870d447d9a3fdefd4b0f60df0, entries=4, sequenceid=5, filesize=5.1 K 2023-07-12 15:04:56,590 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~532 B/532, heapSize ~1.59 KB/1632, currentSize=0 B/0 for 6912f5774859362958fded698274234a in 838ms, sequenceid=5, compaction requested=false 2023-07-12 15:04:56,595 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/data/default/testTruncateWithFailoverAtStep4/6912f5774859362958fded698274234a/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 15:04:56,596 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. 2023-07-12 15:04:56,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 6912f5774859362958fded698274234a: 2023-07-12 15:04:56,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testTruncateWithFailoverAtStep4,b,1689174288036.6912f5774859362958fded698274234a. 2023-07-12 15:04:56,786 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,36649,1689174278222; all regions closed. 2023-07-12 15:04:56,792 DEBUG [RS:0;jenkins-hbase5:36649] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/oldWALs 2023-07-12 15:04:56,792 INFO [RS:0;jenkins-hbase5:36649] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C36649%2C1689174278222.meta:.meta(num 1689174280731) 2023-07-12 15:04:56,795 DEBUG [RS:0;jenkins-hbase5:36649] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/oldWALs 2023-07-12 15:04:56,796 INFO [RS:0;jenkins-hbase5:36649] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C36649%2C1689174278222:(num 1689174279812) 2023-07-12 15:04:56,796 DEBUG [RS:0;jenkins-hbase5:36649] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:56,796 INFO [RS:0;jenkins-hbase5:36649] regionserver.LeaseManager(133): Closed leases 2023-07-12 15:04:56,796 INFO [RS:0;jenkins-hbase5:36649] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase5:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 15:04:56,796 INFO [regionserver/jenkins-hbase5:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:04:56,797 INFO [RS:0;jenkins-hbase5:36649] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:36649 2023-07-12 15:04:56,798 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): regionserver:36649-0x101c5acc8d60002, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase5.apache.org,36649,1689174278222 2023-07-12 15:04:56,798 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:04:56,798 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): regionserver:36649-0x101c5acc8d60002, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:04:56,800 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase5.apache.org,36649,1689174278222] 2023-07-12 15:04:56,801 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase5.apache.org,36649,1689174278222; numProcessing=1 2023-07-12 15:04:56,802 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase5.apache.org,36649,1689174278222 already deleted, retry=false 2023-07-12 15:04:56,802 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase5.apache.org,36649,1689174278222 expired; onlineServers=0 2023-07-12 15:04:56,802 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,41959,1689174278198' ***** 2023-07-12 15:04:56,802 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-12 15:04:56,802 DEBUG [M:1;jenkins-hbase5:41959] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a782f97, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:04:56,802 INFO [M:1;jenkins-hbase5:41959] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,41959,1689174278198 2023-07-12 15:04:56,802 INFO [M:1;jenkins-hbase5:41959] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,41959,1689174278198; all regions closed. 2023-07-12 15:04:56,802 DEBUG [M:1;jenkins-hbase5:41959] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:04:56,802 DEBUG [M:1;jenkins-hbase5:41959] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 15:04:56,802 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 15:04:56,802 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174291676] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174291676,5,FailOnTimeoutGroup] 2023-07-12 15:04:56,802 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174291676] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174291676,5,FailOnTimeoutGroup] 2023-07-12 15:04:56,802 DEBUG [M:1;jenkins-hbase5:41959] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 15:04:56,803 INFO [M:1;jenkins-hbase5:41959] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 15:04:56,803 INFO [M:1;jenkins-hbase5:41959] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 15:04:56,803 INFO [M:1;jenkins-hbase5:41959] hbase.ChoreService(369): Chore service for: master/jenkins-hbase5:0 had [] on shutdown 2023-07-12 15:04:56,804 DEBUG [M:1;jenkins-hbase5:41959] master.HMaster(1512): Stopping service threads 2023-07-12 15:04:56,804 INFO [M:1;jenkins-hbase5:41959] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 15:04:56,804 ERROR [M:1;jenkins-hbase5:41959] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:04:56,804 INFO [M:1;jenkins-hbase5:41959] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:04:56,804 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 15:04:56,804 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:56,804 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:56,804 DEBUG [M:1;jenkins-hbase5:41959] zookeeper.ZKUtil(398): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 15:04:56,805 WARN [M:1;jenkins-hbase5:41959] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 15:04:56,805 INFO [M:1;jenkins-hbase5:41959] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:04:56,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:56,805 INFO [M:1;jenkins-hbase5:41959] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-12 15:04:56,805 DEBUG [M:1;jenkins-hbase5:41959] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:04:56,806 INFO [M:1;jenkins-hbase5:41959] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:56,806 DEBUG [M:1;jenkins-hbase5:41959] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:56,806 DEBUG [M:1;jenkins-hbase5:41959] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:04:56,806 DEBUG [M:1;jenkins-hbase5:41959] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:56,806 INFO [M:1;jenkins-hbase5:41959] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=26.01 KB heapSize=31.70 KB 2023-07-12 15:04:56,900 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): regionserver:36649-0x101c5acc8d60002, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:56,900 INFO [RS:0;jenkins-hbase5:36649] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,36649,1689174278222; zookeeper connection closed. 2023-07-12 15:04:56,900 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): regionserver:36649-0x101c5acc8d60002, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:56,901 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4a577aa0] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4a577aa0 2023-07-12 15:04:56,901 INFO [Listener at localhost/41275] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 1 regionserver(s) complete 2023-07-12 15:04:57,214 INFO [M:1;jenkins-hbase5:41959] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.01 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/025320e5b5d344d5bd4aabb5cea3a2e3 2023-07-12 15:04:57,220 DEBUG [M:1;jenkins-hbase5:41959] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/025320e5b5d344d5bd4aabb5cea3a2e3 as hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/025320e5b5d344d5bd4aabb5cea3a2e3 2023-07-12 15:04:57,225 INFO [M:1;jenkins-hbase5:41959] regionserver.HStore(1080): Added hdfs://localhost:40575/user/jenkins/test-data/98f497a8-365a-6c50-003a-2d54701de941/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/025320e5b5d344d5bd4aabb5cea3a2e3, entries=9, sequenceid=253, filesize=5.4 K 2023-07-12 15:04:57,226 INFO [M:1;jenkins-hbase5:41959] regionserver.HRegion(2948): Finished flush of dataSize ~26.01 KB/26634, heapSize ~31.68 KB/32440, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 420ms, sequenceid=253, compaction requested=false 2023-07-12 15:04:57,228 INFO [M:1;jenkins-hbase5:41959] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:57,228 DEBUG [M:1;jenkins-hbase5:41959] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:57,231 INFO [M:1;jenkins-hbase5:41959] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 15:04:57,231 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:04:57,232 INFO [M:1;jenkins-hbase5:41959] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:41959 2023-07-12 15:04:57,234 DEBUG [M:1;jenkins-hbase5:41959] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase5.apache.org,41959,1689174278198 already deleted, retry=false 2023-07-12 15:04:57,336 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:57,336 DEBUG [Listener at localhost/41275-EventThread] zookeeper.ZKWatcher(600): master:41959-0x101c5acc8d60001, quorum=127.0.0.1:54427, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:04:57,336 INFO [M:1;jenkins-hbase5:41959] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,41959,1689174278198; zookeeper connection closed. 2023-07-12 15:04:57,337 WARN [Listener at localhost/41275] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 15:04:57,340 INFO [Listener at localhost/41275] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 15:04:57,446 WARN [BP-19536874-172.31.10.131-1689174277344 heartbeating to localhost/127.0.0.1:40575] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 15:04:57,446 WARN [BP-19536874-172.31.10.131-1689174277344 heartbeating to localhost/127.0.0.1:40575] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-19536874-172.31.10.131-1689174277344 (Datanode Uuid 2e1fc74d-8826-44de-af2b-d81e3f1a7569) service to localhost/127.0.0.1:40575 2023-07-12 15:04:57,447 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/cluster_a0c31478-57e2-8297-176a-88a31018afde/dfs/data/data1/current/BP-19536874-172.31.10.131-1689174277344] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 15:04:57,447 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/cluster_a0c31478-57e2-8297-176a-88a31018afde/dfs/data/data2/current/BP-19536874-172.31.10.131-1689174277344] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 15:04:57,460 INFO [Listener at localhost/41275] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 15:04:57,538 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-12 15:04:57,579 INFO [Listener at localhost/41275] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 15:04:57,589 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-07-12 15:04:57,600 INFO [Listener at localhost/41275] hbase.ResourceChecker(175): after: master.procedure.TestMasterFailoverWithProcedures#testTruncateWithFailover Thread=125 (was 102) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/41275 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.3@localhost:40575 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (593222724) connection to localhost/127.0.0.1:40575 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: LeaseRenewer:jenkins@localhost:40575 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (593222724) connection to localhost/127.0.0.1:40575 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-13-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (593222724) connection to localhost/127.0.0.1:40575 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) - Thread LEAK? -, OpenFileDescriptor=487 (was 450) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=184 (was 237), ProcessCount=161 (was 161), AvailableMemoryMB=7722 (was 7911) 2023-07-12 15:04:57,609 INFO [Listener at localhost/41275] hbase.ResourceChecker(147): before: master.procedure.TestMasterFailoverWithProcedures#testEnableTableWithFailover Thread=125, OpenFileDescriptor=487, MaxFileDescriptor=60000, SystemLoadAverage=184, ProcessCount=161, AvailableMemoryMB=7722 2023-07-12 15:04:57,609 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=2, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-12 15:04:57,609 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/hadoop.log.dir so I do NOT create it in target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0 2023-07-12 15:04:57,609 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c57d4f25-4578-d1d9-5797-a672e6fb7232/hadoop.tmp.dir so I do NOT create it in target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0 2023-07-12 15:04:57,609 INFO [Listener at localhost/41275] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/cluster_de344728-3c1b-401b-1994-b1d21d451bc9, deleteOnExit=true 2023-07-12 15:04:57,610 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-12 15:04:57,610 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/test.cache.data in system properties and HBase conf 2023-07-12 15:04:57,610 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 15:04:57,610 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/hadoop.log.dir in system properties and HBase conf 2023-07-12 15:04:57,610 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 15:04:57,610 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 15:04:57,610 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 15:04:57,610 DEBUG [Listener at localhost/41275] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 15:04:57,611 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 15:04:57,611 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 15:04:57,611 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 15:04:57,611 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 15:04:57,611 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 15:04:57,611 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 15:04:57,611 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 15:04:57,611 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 15:04:57,611 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 15:04:57,611 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/nfs.dump.dir in system properties and HBase conf 2023-07-12 15:04:57,612 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/java.io.tmpdir in system properties and HBase conf 2023-07-12 15:04:57,612 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 15:04:57,612 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 15:04:57,612 INFO [Listener at localhost/41275] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 15:04:57,616 WARN [Listener at localhost/41275] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 15:04:57,616 WARN [Listener at localhost/41275] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 15:04:57,654 WARN [Listener at localhost/41275] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 15:04:57,656 INFO [Listener at localhost/41275] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 15:04:57,661 INFO [Listener at localhost/41275] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/java.io.tmpdir/Jetty_localhost_36463_hdfs____.nrs52g/webapp 2023-07-12 15:04:57,754 INFO [Listener at localhost/41275] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36463 2023-07-12 15:04:57,758 WARN [Listener at localhost/41275] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 15:04:57,758 WARN [Listener at localhost/41275] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 15:04:57,787 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(152): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 15:04:57,795 WARN [Listener at localhost/40551] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 15:04:57,805 WARN [Listener at localhost/40551] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 15:04:57,807 WARN [Listener at localhost/40551] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 15:04:57,808 INFO [Listener at localhost/40551] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 15:04:57,813 INFO [Listener at localhost/40551] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/java.io.tmpdir/Jetty_localhost_37131_datanode____.2beyb2/webapp 2023-07-12 15:04:57,908 INFO [Listener at localhost/40551] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37131 2023-07-12 15:04:57,915 WARN [Listener at localhost/44677] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 15:04:58,021 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x35843cf1ee3af0cc: Processing first storage report for DS-8861a867-3cf6-47c1-b2f9-7ab4a25576e7 from datanode 592f0b78-afac-481f-be23-d1838c16fb33 2023-07-12 15:04:58,021 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x35843cf1ee3af0cc: from storage DS-8861a867-3cf6-47c1-b2f9-7ab4a25576e7 node DatanodeRegistration(127.0.0.1:36041, datanodeUuid=592f0b78-afac-481f-be23-d1838c16fb33, infoPort=44995, infoSecurePort=0, ipcPort=44677, storageInfo=lv=-57;cid=testClusterID;nsid=366827402;c=1689174297619), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 15:04:58,021 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x35843cf1ee3af0cc: Processing first storage report for DS-f933a034-15b0-4565-a7e4-f16a5b530448 from datanode 592f0b78-afac-481f-be23-d1838c16fb33 2023-07-12 15:04:58,021 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x35843cf1ee3af0cc: from storage DS-f933a034-15b0-4565-a7e4-f16a5b530448 node DatanodeRegistration(127.0.0.1:36041, datanodeUuid=592f0b78-afac-481f-be23-d1838c16fb33, infoPort=44995, infoSecurePort=0, ipcPort=44677, storageInfo=lv=-57;cid=testClusterID;nsid=366827402;c=1689174297619), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 15:04:58,123 DEBUG [Listener at localhost/44677] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0 2023-07-12 15:04:58,126 INFO [Listener at localhost/44677] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/cluster_de344728-3c1b-401b-1994-b1d21d451bc9/zookeeper_0, clientPort=51235, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/cluster_de344728-3c1b-401b-1994-b1d21d451bc9/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/cluster_de344728-3c1b-401b-1994-b1d21d451bc9/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 15:04:58,127 INFO [Listener at localhost/44677] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=51235 2023-07-12 15:04:58,127 INFO [Listener at localhost/44677] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:58,128 INFO [Listener at localhost/44677] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:58,538 INFO [Listener at localhost/44677] util.FSUtils(471): Created version file at hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919 with version=8 2023-07-12 15:04:58,539 INFO [Listener at localhost/44677] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost:46679/user/jenkins/test-data/6379c6d9-186c-5c91-9fa5-c5d4330d3411/hbase-staging 2023-07-12 15:04:58,540 INFO [Listener at localhost/44677] client.ConnectionUtils(127): master/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:04:58,541 INFO [Listener at localhost/44677] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:58,541 INFO [Listener at localhost/44677] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:58,541 INFO [Listener at localhost/44677] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:04:58,541 INFO [Listener at localhost/44677] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:58,541 INFO [Listener at localhost/44677] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:04:58,541 INFO [Listener at localhost/44677] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:04:58,542 INFO [Listener at localhost/44677] ipc.NettyRpcServer(120): Bind to /172.31.10.131:34909 2023-07-12 15:04:58,542 INFO [Listener at localhost/44677] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:58,543 INFO [Listener at localhost/44677] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:58,544 INFO [Listener at localhost/44677] zookeeper.RecoverableZooKeeper(93): Process identifier=master:34909 connecting to ZooKeeper ensemble=127.0.0.1:51235 2023-07-12 15:04:58,552 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:349090x0, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:04:58,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:34909-0x101c5ad18690000 connected 2023-07-12 15:04:58,565 DEBUG [Listener at localhost/44677] zookeeper.ZKUtil(164): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:58,565 DEBUG [Listener at localhost/44677] zookeeper.ZKUtil(164): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:58,566 DEBUG [Listener at localhost/44677] zookeeper.ZKUtil(164): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:04:58,566 DEBUG [Listener at localhost/44677] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34909 2023-07-12 15:04:58,566 DEBUG [Listener at localhost/44677] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34909 2023-07-12 15:04:58,566 DEBUG [Listener at localhost/44677] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34909 2023-07-12 15:04:58,567 DEBUG [Listener at localhost/44677] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34909 2023-07-12 15:04:58,567 DEBUG [Listener at localhost/44677] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34909 2023-07-12 15:04:58,567 INFO [Listener at localhost/44677] master.HMaster(444): hbase.rootdir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919, hbase.cluster.distributed=false 2023-07-12 15:04:58,569 INFO [Listener at localhost/44677] client.ConnectionUtils(127): master/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:04:58,569 INFO [Listener at localhost/44677] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:58,569 INFO [Listener at localhost/44677] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:58,569 INFO [Listener at localhost/44677] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:04:58,569 INFO [Listener at localhost/44677] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:58,569 INFO [Listener at localhost/44677] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:04:58,569 INFO [Listener at localhost/44677] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:04:58,570 INFO [Listener at localhost/44677] ipc.NettyRpcServer(120): Bind to /172.31.10.131:46407 2023-07-12 15:04:58,570 INFO [Listener at localhost/44677] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:58,571 INFO [Listener at localhost/44677] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:58,572 INFO [Listener at localhost/44677] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46407 connecting to ZooKeeper ensemble=127.0.0.1:51235 2023-07-12 15:04:58,575 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:464070x0, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:04:58,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:46407-0x101c5ad18690001 connected 2023-07-12 15:04:58,578 DEBUG [Listener at localhost/44677] zookeeper.ZKUtil(164): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:58,578 DEBUG [Listener at localhost/44677] zookeeper.ZKUtil(164): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:58,579 DEBUG [Listener at localhost/44677] zookeeper.ZKUtil(164): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:04:58,579 DEBUG [Listener at localhost/44677] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46407 2023-07-12 15:04:58,579 DEBUG [Listener at localhost/44677] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46407 2023-07-12 15:04:58,579 DEBUG [Listener at localhost/44677] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46407 2023-07-12 15:04:58,580 DEBUG [Listener at localhost/44677] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46407 2023-07-12 15:04:58,580 DEBUG [Listener at localhost/44677] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46407 2023-07-12 15:04:58,580 INFO [Listener at localhost/44677] master.HMaster(444): hbase.rootdir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919, hbase.cluster.distributed=false 2023-07-12 15:04:58,592 INFO [Listener at localhost/44677] client.ConnectionUtils(127): regionserver/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:04:58,592 INFO [Listener at localhost/44677] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:58,592 INFO [Listener at localhost/44677] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:58,592 INFO [Listener at localhost/44677] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:04:58,592 INFO [Listener at localhost/44677] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:04:58,593 INFO [Listener at localhost/44677] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:04:58,593 INFO [Listener at localhost/44677] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:04:58,593 INFO [Listener at localhost/44677] ipc.NettyRpcServer(120): Bind to /172.31.10.131:35497 2023-07-12 15:04:58,593 INFO [Listener at localhost/44677] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 15:04:58,594 DEBUG [Listener at localhost/44677] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 15:04:58,595 INFO [Listener at localhost/44677] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:58,595 INFO [Listener at localhost/44677] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:58,596 INFO [Listener at localhost/44677] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35497 connecting to ZooKeeper ensemble=127.0.0.1:51235 2023-07-12 15:04:58,599 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): regionserver:354970x0, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:04:58,600 DEBUG [Listener at localhost/44677] zookeeper.ZKUtil(164): regionserver:354970x0, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:04:58,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:35497-0x101c5ad18690002 connected 2023-07-12 15:04:58,600 DEBUG [Listener at localhost/44677] zookeeper.ZKUtil(164): regionserver:35497-0x101c5ad18690002, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:04:58,601 DEBUG [Listener at localhost/44677] zookeeper.ZKUtil(164): regionserver:35497-0x101c5ad18690002, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:04:58,602 DEBUG [Listener at localhost/44677] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35497 2023-07-12 15:04:58,602 DEBUG [Listener at localhost/44677] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35497 2023-07-12 15:04:58,602 DEBUG [Listener at localhost/44677] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35497 2023-07-12 15:04:58,602 DEBUG [Listener at localhost/44677] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35497 2023-07-12 15:04:58,603 DEBUG [Listener at localhost/44677] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35497 2023-07-12 15:04:58,604 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase5.apache.org,34909,1689174298540 2023-07-12 15:04:58,604 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase5.apache.org,46407,1689174298568 2023-07-12 15:04:58,605 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:58,605 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:58,607 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase5.apache.org,34909,1689174298540 2023-07-12 15:04:58,607 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase5.apache.org,46407,1689174298568 2023-07-12 15:04:58,608 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:58,608 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:58,608 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:58,608 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): regionserver:35497-0x101c5ad18690002, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:04:58,608 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:58,609 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:58,611 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:58,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:58,611 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase5.apache.org,34909,1689174298540 from backup master directory 2023-07-12 15:04:58,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:04:58,611 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase5.apache.org,34909,1689174298540; waiting to become the next active master 2023-07-12 15:04:58,612 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase5.apache.org,34909,1689174298540 2023-07-12 15:04:58,612 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:58,612 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:04:58,612 WARN [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:04:58,612 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase5.apache.org,34909,1689174298540 2023-07-12 15:04:59,024 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/hbase.id with ID: c7a058ac-e0fb-44ee-918b-00820fa6abd0 2023-07-12 15:04:59,038 INFO [master/jenkins-hbase5:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:04:59,042 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:59,042 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:59,053 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0e1e3eaa to 127.0.0.1:51235 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:59,061 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50fff9d7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:59,061 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:59,061 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 15:04:59,063 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:59,065 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store-tmp 2023-07-12 15:04:59,473 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:59,473 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:04:59,473 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:59,473 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:59,473 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:04:59,473 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:59,473 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:04:59,473 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:59,474 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/WALs/jenkins-hbase5.apache.org,34909,1689174298540 2023-07-12 15:04:59,477 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C34909%2C1689174298540, suffix=, logDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/WALs/jenkins-hbase5.apache.org,34909,1689174298540, archiveDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/oldWALs, maxLogs=10 2023-07-12 15:04:59,492 DEBUG [RS-EventLoopGroup-17-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36041,DS-8861a867-3cf6-47c1-b2f9-7ab4a25576e7,DISK] 2023-07-12 15:04:59,494 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/WALs/jenkins-hbase5.apache.org,34909,1689174298540/jenkins-hbase5.apache.org%2C34909%2C1689174298540.1689174299477 2023-07-12 15:04:59,494 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36041,DS-8861a867-3cf6-47c1-b2f9-7ab4a25576e7,DISK]] 2023-07-12 15:04:59,495 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:04:59,495 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:04:59,495 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:59,495 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:59,496 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:59,497 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 15:04:59,498 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 15:04:59,498 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:04:59,499 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:59,500 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:59,502 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:04:59,504 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:04:59,504 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69779705, jitterRate=0.03979863226413727}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:04:59,504 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:04:59,504 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 15:04:59,505 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:04:59,505 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:04:59,505 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:04:59,506 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:04:59,506 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-07-12 15:04:59,506 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 15:04:59,507 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 15:04:59,508 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 15:04:59,518 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 15:04:59,518 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 15:04:59,518 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 15:04:59,518 INFO [master/jenkins-hbase5:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 15:04:59,519 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 15:04:59,521 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:59,521 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:59,521 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 15:04:59,521 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 15:04:59,522 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 15:04:59,523 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:59,523 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): regionserver:35497-0x101c5ad18690002, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:59,523 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:59,523 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:04:59,523 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:59,524 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase5.apache.org,34909,1689174298540, sessionid=0x101c5ad18690000, setting cluster-up flag (Was=false) 2023-07-12 15:04:59,526 INFO [M:1;jenkins-hbase5:46407] regionserver.HRegionServer(951): ClusterId : c7a058ac-e0fb-44ee-918b-00820fa6abd0 2023-07-12 15:04:59,528 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:59,528 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:59,532 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 15:04:59,533 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,34909,1689174298540 2023-07-12 15:04:59,536 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:59,536 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:04:59,540 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 15:04:59,541 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,34909,1689174298540 2023-07-12 15:04:59,541 WARN [master/jenkins-hbase5:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.hbase-snapshot/.tmp 2023-07-12 15:04:59,543 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:59,543 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:59,543 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:59,543 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:59,543 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:04:59,543 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase5:0, corePoolSize=10, maxPoolSize=10 2023-07-12 15:04:59,543 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:59,543 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:04:59,543 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:59,545 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174329545 2023-07-12 15:04:59,545 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 15:04:59,546 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 15:04:59,546 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 15:04:59,546 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 15:04:59,546 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 15:04:59,546 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 15:04:59,546 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:59,547 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 15:04:59,548 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 15:04:59,548 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 15:04:59,548 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 15:04:59,548 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 15:04:59,548 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 15:04:59,548 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 15:04:59,548 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174299548,5,FailOnTimeoutGroup] 2023-07-12 15:04:59,549 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 15:04:59,553 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174299548,5,FailOnTimeoutGroup] 2023-07-12 15:04:59,553 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:59,553 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 15:04:59,553 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:59,553 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:59,606 INFO [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(951): ClusterId : c7a058ac-e0fb-44ee-918b-00820fa6abd0 2023-07-12 15:04:59,606 DEBUG [RS:0;jenkins-hbase5:35497] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 15:04:59,608 DEBUG [RS:0;jenkins-hbase5:35497] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 15:04:59,608 DEBUG [RS:0;jenkins-hbase5:35497] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 15:04:59,610 DEBUG [RS:0;jenkins-hbase5:35497] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 15:04:59,611 DEBUG [RS:0;jenkins-hbase5:35497] zookeeper.ReadOnlyZKClient(139): Connect 0x7fda9f2e to 127.0.0.1:51235 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:04:59,617 DEBUG [RS:0;jenkins-hbase5:35497] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@711d6ea5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:04:59,617 DEBUG [RS:0;jenkins-hbase5:35497] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7522f45e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:04:59,625 DEBUG [RS:0;jenkins-hbase5:35497] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase5:35497 2023-07-12 15:04:59,626 INFO [RS:0;jenkins-hbase5:35497] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 15:04:59,626 INFO [RS:0;jenkins-hbase5:35497] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 15:04:59,626 DEBUG [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 15:04:59,626 INFO [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase5.apache.org,34909,1689174298540 with isa=jenkins-hbase5.apache.org/172.31.10.131:35497, startcode=1689174298592 2023-07-12 15:04:59,626 DEBUG [RS:0;jenkins-hbase5:35497] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 15:04:59,629 INFO [RS-EventLoopGroup-15-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:55847, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 15:04:59,630 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34909] master.ServerManager(394): Registering regionserver=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:04:59,630 DEBUG [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919 2023-07-12 15:04:59,630 DEBUG [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:40551 2023-07-12 15:04:59,630 DEBUG [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 15:04:59,632 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:04:59,632 DEBUG [RS:0;jenkins-hbase5:35497] zookeeper.ZKUtil(162): regionserver:35497-0x101c5ad18690002, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:04:59,632 WARN [RS:0;jenkins-hbase5:35497] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:04:59,633 INFO [RS:0;jenkins-hbase5:35497] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:04:59,633 DEBUG [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(1948): logDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/WALs/jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:04:59,633 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase5.apache.org,35497,1689174298592] 2023-07-12 15:04:59,636 DEBUG [RS:0;jenkins-hbase5:35497] zookeeper.ZKUtil(162): regionserver:35497-0x101c5ad18690002, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:04:59,637 DEBUG [RS:0;jenkins-hbase5:35497] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 15:04:59,637 INFO [RS:0;jenkins-hbase5:35497] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 15:04:59,638 INFO [RS:0;jenkins-hbase5:35497] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 15:04:59,638 INFO [RS:0;jenkins-hbase5:35497] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:04:59,639 INFO [RS:0;jenkins-hbase5:35497] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:59,639 INFO [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 15:04:59,640 INFO [RS:0;jenkins-hbase5:35497] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:59,640 DEBUG [RS:0;jenkins-hbase5:35497] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:59,640 DEBUG [RS:0;jenkins-hbase5:35497] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:59,640 DEBUG [RS:0;jenkins-hbase5:35497] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:59,640 DEBUG [RS:0;jenkins-hbase5:35497] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:59,640 DEBUG [RS:0;jenkins-hbase5:35497] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:59,640 DEBUG [RS:0;jenkins-hbase5:35497] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:04:59,640 DEBUG [RS:0;jenkins-hbase5:35497] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:59,640 DEBUG [RS:0;jenkins-hbase5:35497] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:59,640 DEBUG [RS:0;jenkins-hbase5:35497] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:59,641 DEBUG [RS:0;jenkins-hbase5:35497] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:04:59,641 INFO [RS:0;jenkins-hbase5:35497] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:59,641 INFO [RS:0;jenkins-hbase5:35497] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:59,641 INFO [RS:0;jenkins-hbase5:35497] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:59,657 INFO [RS:0;jenkins-hbase5:35497] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 15:04:59,657 INFO [RS:0;jenkins-hbase5:35497] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,35497,1689174298592-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:04:59,668 INFO [RS:0;jenkins-hbase5:35497] regionserver.Replication(203): jenkins-hbase5.apache.org,35497,1689174298592 started 2023-07-12 15:04:59,668 INFO [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(1637): Serving as jenkins-hbase5.apache.org,35497,1689174298592, RpcServer on jenkins-hbase5.apache.org/172.31.10.131:35497, sessionid=0x101c5ad18690002 2023-07-12 15:04:59,668 DEBUG [RS:0;jenkins-hbase5:35497] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 15:04:59,668 DEBUG [RS:0;jenkins-hbase5:35497] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:04:59,668 DEBUG [RS:0;jenkins-hbase5:35497] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,35497,1689174298592' 2023-07-12 15:04:59,668 DEBUG [RS:0;jenkins-hbase5:35497] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:04:59,669 DEBUG [RS:0;jenkins-hbase5:35497] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:04:59,669 DEBUG [RS:0;jenkins-hbase5:35497] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 15:04:59,670 DEBUG [RS:0;jenkins-hbase5:35497] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 15:04:59,670 DEBUG [RS:0;jenkins-hbase5:35497] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:04:59,670 DEBUG [RS:0;jenkins-hbase5:35497] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,35497,1689174298592' 2023-07-12 15:04:59,670 DEBUG [RS:0;jenkins-hbase5:35497] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 15:04:59,670 DEBUG [RS:0;jenkins-hbase5:35497] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 15:04:59,670 DEBUG [RS:0;jenkins-hbase5:35497] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 15:04:59,670 INFO [RS:0;jenkins-hbase5:35497] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 15:04:59,670 INFO [RS:0;jenkins-hbase5:35497] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 15:04:59,772 INFO [RS:0;jenkins-hbase5:35497] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C35497%2C1689174298592, suffix=, logDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/WALs/jenkins-hbase5.apache.org,35497,1689174298592, archiveDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/oldWALs, maxLogs=32 2023-07-12 15:04:59,787 DEBUG [RS-EventLoopGroup-17-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36041,DS-8861a867-3cf6-47c1-b2f9-7ab4a25576e7,DISK] 2023-07-12 15:04:59,789 INFO [RS:0;jenkins-hbase5:35497] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/WALs/jenkins-hbase5.apache.org,35497,1689174298592/jenkins-hbase5.apache.org%2C35497%2C1689174298592.1689174299773 2023-07-12 15:04:59,789 DEBUG [RS:0;jenkins-hbase5:35497] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36041,DS-8861a867-3cf6-47c1-b2f9-7ab4a25576e7,DISK]] 2023-07-12 15:04:59,960 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:59,960 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 15:04:59,961 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919 2023-07-12 15:05:00,367 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:00,374 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 15:05:00,375 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/info 2023-07-12 15:05:00,375 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 15:05:00,376 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:00,376 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 15:05:00,384 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:05:00,384 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 15:05:00,384 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:00,384 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 15:05:00,386 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/table 2023-07-12 15:05:00,386 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 15:05:00,386 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:00,387 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740 2023-07-12 15:05:00,387 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740 2023-07-12 15:05:00,389 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 15:05:00,390 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 15:05:00,392 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:05:00,392 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63362619, jitterRate=-0.05582340061664581}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 15:05:00,392 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 15:05:00,392 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 15:05:00,392 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 15:05:00,392 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 15:05:00,392 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 15:05:00,392 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 15:05:00,393 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 15:05:00,393 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 15:05:00,394 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 15:05:00,394 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 15:05:00,394 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 15:05:00,395 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 15:05:00,396 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 15:05:00,546 DEBUG [jenkins-hbase5:34909] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 15:05:00,547 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,35497,1689174298592, state=OPENING 2023-07-12 15:05:00,549 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 15:05:00,551 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:05:00,551 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:05:00,552 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:05:00,552 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:05:00,552 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,35497,1689174298592}] 2023-07-12 15:05:00,704 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:00,705 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 15:05:00,707 INFO [RS-EventLoopGroup-17-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:54136, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 15:05:00,711 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 15:05:00,711 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:05:00,712 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C35497%2C1689174298592.meta, suffix=.meta, logDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/WALs/jenkins-hbase5.apache.org,35497,1689174298592, archiveDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/oldWALs, maxLogs=32 2023-07-12 15:05:00,729 DEBUG [RS-EventLoopGroup-17-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36041,DS-8861a867-3cf6-47c1-b2f9-7ab4a25576e7,DISK] 2023-07-12 15:05:00,731 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/WALs/jenkins-hbase5.apache.org,35497,1689174298592/jenkins-hbase5.apache.org%2C35497%2C1689174298592.meta.1689174300713.meta 2023-07-12 15:05:00,731 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36041,DS-8861a867-3cf6-47c1-b2f9-7ab4a25576e7,DISK]] 2023-07-12 15:05:00,731 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:05:00,731 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 15:05:00,731 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 15:05:00,731 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 15:05:00,732 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 15:05:00,732 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:00,732 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 15:05:00,732 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 15:05:00,733 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 15:05:00,734 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/info 2023-07-12 15:05:00,734 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/info 2023-07-12 15:05:00,734 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 15:05:00,735 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:00,735 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 15:05:00,736 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:05:00,736 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:05:00,737 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 15:05:00,737 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:00,737 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 15:05:00,738 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/table 2023-07-12 15:05:00,738 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/table 2023-07-12 15:05:00,739 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 15:05:00,739 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:00,740 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740 2023-07-12 15:05:00,741 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740 2023-07-12 15:05:00,743 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 15:05:00,744 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 15:05:00,745 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65579866, jitterRate=-0.02278384566307068}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 15:05:00,745 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 15:05:00,746 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689174300704 2023-07-12 15:05:00,749 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 15:05:00,750 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 15:05:00,750 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,35497,1689174298592, state=OPEN 2023-07-12 15:05:00,754 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 15:05:00,754 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 15:05:00,754 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:05:00,754 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:05:00,756 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 15:05:00,756 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,35497,1689174298592 in 203 msec 2023-07-12 15:05:00,758 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 15:05:00,758 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 362 msec 2023-07-12 15:05:00,760 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.2160 sec 2023-07-12 15:05:00,760 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689174300760, completionTime=-1 2023-07-12 15:05:00,760 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 15:05:00,760 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:05:00,763 DEBUG [hconnection-0x1e3ff020-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:05:00,765 INFO [RS-EventLoopGroup-17-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:54144, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:05:00,766 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:05:00,767 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174360767 2023-07-12 15:05:00,767 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174420767 2023-07-12 15:05:00,767 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-07-12 15:05:00,773 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,34909,1689174298540-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:05:00,773 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,34909,1689174298540-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:05:00,773 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,34909,1689174298540-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:05:00,773 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase5:34909, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:05:00,773 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 15:05:00,773 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 15:05:00,774 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 15:05:00,775 DEBUG [master/jenkins-hbase5:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 15:05:00,775 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 15:05:00,776 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:05:00,777 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:05:00,779 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/hbase/namespace/b4ef2da9a8e7a43cdb82478409694269 2023-07-12 15:05:00,779 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/hbase/namespace/b4ef2da9a8e7a43cdb82478409694269 empty. 2023-07-12 15:05:00,780 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/hbase/namespace/b4ef2da9a8e7a43cdb82478409694269 2023-07-12 15:05:00,780 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 15:05:00,795 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 15:05:00,796 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => b4ef2da9a8e7a43cdb82478409694269, NAME => 'hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp 2023-07-12 15:05:00,807 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:00,807 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing b4ef2da9a8e7a43cdb82478409694269, disabling compactions & flushes 2023-07-12 15:05:00,807 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269. 2023-07-12 15:05:00,807 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269. 2023-07-12 15:05:00,807 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269. after waiting 0 ms 2023-07-12 15:05:00,807 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269. 2023-07-12 15:05:00,807 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269. 2023-07-12 15:05:00,807 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for b4ef2da9a8e7a43cdb82478409694269: 2023-07-12 15:05:00,809 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:05:00,810 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689174300810"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174300810"}]},"ts":"1689174300810"} 2023-07-12 15:05:00,812 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 15:05:00,813 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:05:00,813 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174300813"}]},"ts":"1689174300813"} 2023-07-12 15:05:00,814 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 15:05:00,818 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=b4ef2da9a8e7a43cdb82478409694269, ASSIGN}] 2023-07-12 15:05:00,820 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=b4ef2da9a8e7a43cdb82478409694269, ASSIGN 2023-07-12 15:05:00,820 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=b4ef2da9a8e7a43cdb82478409694269, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,35497,1689174298592; forceNewPlan=false, retain=false 2023-07-12 15:05:00,971 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=b4ef2da9a8e7a43cdb82478409694269, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:00,972 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689174300971"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174300971"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174300971"}]},"ts":"1689174300971"} 2023-07-12 15:05:00,974 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure b4ef2da9a8e7a43cdb82478409694269, server=jenkins-hbase5.apache.org,35497,1689174298592}] 2023-07-12 15:05:01,148 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269. 2023-07-12 15:05:01,148 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b4ef2da9a8e7a43cdb82478409694269, NAME => 'hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:05:01,148 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace b4ef2da9a8e7a43cdb82478409694269 2023-07-12 15:05:01,148 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:01,149 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for b4ef2da9a8e7a43cdb82478409694269 2023-07-12 15:05:01,149 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for b4ef2da9a8e7a43cdb82478409694269 2023-07-12 15:05:01,151 INFO [StoreOpener-b4ef2da9a8e7a43cdb82478409694269-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region b4ef2da9a8e7a43cdb82478409694269 2023-07-12 15:05:01,152 DEBUG [StoreOpener-b4ef2da9a8e7a43cdb82478409694269-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/namespace/b4ef2da9a8e7a43cdb82478409694269/info 2023-07-12 15:05:01,153 DEBUG [StoreOpener-b4ef2da9a8e7a43cdb82478409694269-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/namespace/b4ef2da9a8e7a43cdb82478409694269/info 2023-07-12 15:05:01,153 INFO [StoreOpener-b4ef2da9a8e7a43cdb82478409694269-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b4ef2da9a8e7a43cdb82478409694269 columnFamilyName info 2023-07-12 15:05:01,153 INFO [StoreOpener-b4ef2da9a8e7a43cdb82478409694269-1] regionserver.HStore(310): Store=b4ef2da9a8e7a43cdb82478409694269/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:01,154 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/namespace/b4ef2da9a8e7a43cdb82478409694269 2023-07-12 15:05:01,155 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/namespace/b4ef2da9a8e7a43cdb82478409694269 2023-07-12 15:05:01,158 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for b4ef2da9a8e7a43cdb82478409694269 2023-07-12 15:05:01,163 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/namespace/b4ef2da9a8e7a43cdb82478409694269/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:05:01,164 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened b4ef2da9a8e7a43cdb82478409694269; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68357987, jitterRate=0.018613383173942566}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:05:01,164 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for b4ef2da9a8e7a43cdb82478409694269: 2023-07-12 15:05:01,165 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269., pid=6, masterSystemTime=1689174301126 2023-07-12 15:05:01,167 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269. 2023-07-12 15:05:01,168 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269. 2023-07-12 15:05:01,168 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=b4ef2da9a8e7a43cdb82478409694269, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:01,168 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689174301168"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174301168"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174301168"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174301168"}]},"ts":"1689174301168"} 2023-07-12 15:05:01,171 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 15:05:01,171 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure b4ef2da9a8e7a43cdb82478409694269, server=jenkins-hbase5.apache.org,35497,1689174298592 in 196 msec 2023-07-12 15:05:01,172 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 15:05:01,172 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=b4ef2da9a8e7a43cdb82478409694269, ASSIGN in 353 msec 2023-07-12 15:05:01,173 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 15:05:01,173 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174301173"}]},"ts":"1689174301173"} 2023-07-12 15:05:01,174 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 15:05:01,176 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 15:05:01,176 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 15:05:01,178 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 403 msec 2023-07-12 15:05:01,178 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:05:01,178 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:05:01,178 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:05:01,181 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 15:05:01,189 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:05:01,192 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 10 msec 2023-07-12 15:05:01,204 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:05:01,217 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:05:01,219 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-07-12 15:05:01,230 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 15:05:01,232 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 15:05:01,232 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.620sec 2023-07-12 15:05:01,232 INFO [master/jenkins-hbase5:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 15:05:01,232 INFO [master/jenkins-hbase5:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 15:05:01,232 INFO [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 15:05:01,232 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,34909,1689174298540-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 15:05:01,232 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,34909,1689174298540-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 15:05:01,234 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 15:05:01,309 DEBUG [Listener at localhost/44677] zookeeper.ReadOnlyZKClient(139): Connect 0x05948539 to 127.0.0.1:51235 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:05:01,314 DEBUG [Listener at localhost/44677] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6aedc898, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:05:01,316 DEBUG [hconnection-0x216d1332-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:05:01,321 INFO [RS-EventLoopGroup-17-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:54160, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:05:01,322 INFO [Listener at localhost/44677] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase5.apache.org,34909,1689174298540 2023-07-12 15:05:01,322 WARN [Listener at localhost/44677] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-12 15:05:01,326 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:01,328 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:05:01,328 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:05:01,333 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a 2023-07-12 15:05:01,333 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:01,333 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888 2023-07-12 15:05:01,333 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:01,334 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a empty. 2023-07-12 15:05:01,334 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f empty. 2023-07-12 15:05:01,334 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f empty. 2023-07-12 15:05:01,334 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888 empty. 2023-07-12 15:05:01,343 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:01,343 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:01,343 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a 2023-07-12 15:05:01,343 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888 2023-07-12 15:05:01,344 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testEnableTableWithFailoverAtStep3 regions 2023-07-12 15:05:01,759 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp/data/default/testEnableTableWithFailoverAtStep3/.tabledesc/.tableinfo.0000000001 2023-07-12 15:05:01,761 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(7675): creating {ENCODED => 365408da311d11f25467a308428c748a, NAME => 'testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a.', STARTKEY => '', ENDKEY => 'a'}, tableDescriptor='testEnableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp 2023-07-12 15:05:01,761 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(7675): creating {ENCODED => bad388b7482b4b27d67bc96cd80a265f, NAME => 'testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f.', STARTKEY => 'b', ENDKEY => 'c'}, tableDescriptor='testEnableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp 2023-07-12 15:05:01,762 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(7675): creating {ENCODED => d50f055275b35092de8260c021ea4888, NAME => 'testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888.', STARTKEY => 'a', ENDKEY => 'b'}, tableDescriptor='testEnableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp 2023-07-12 15:05:01,783 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:01,783 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1604): Closing bad388b7482b4b27d67bc96cd80a265f, disabling compactions & flushes 2023-07-12 15:05:01,784 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:01,784 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:01,784 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. after waiting 0 ms 2023-07-12 15:05:01,784 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:01,784 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:01,784 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1558): Region close journal for bad388b7482b4b27d67bc96cd80a265f: 2023-07-12 15:05:01,784 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(7675): creating {ENCODED => ffe3dd29195363fb366cf4b7acdf928f, NAME => 'testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f.', STARTKEY => 'c', ENDKEY => ''}, tableDescriptor='testEnableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.tmp 2023-07-12 15:05:01,785 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:01,785 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1604): Closing d50f055275b35092de8260c021ea4888, disabling compactions & flushes 2023-07-12 15:05:01,785 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:01,785 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:01,785 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. after waiting 0 ms 2023-07-12 15:05:01,786 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:01,786 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:01,786 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1558): Region close journal for d50f055275b35092de8260c021ea4888: 2023-07-12 15:05:02,182 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:02,183 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1604): Closing 365408da311d11f25467a308428c748a, disabling compactions & flushes 2023-07-12 15:05:02,183 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:02,183 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:02,183 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. after waiting 0 ms 2023-07-12 15:05:02,183 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:02,183 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:02,183 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1558): Region close journal for 365408da311d11f25467a308428c748a: 2023-07-12 15:05:02,192 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:02,192 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1604): Closing ffe3dd29195363fb366cf4b7acdf928f, disabling compactions & flushes 2023-07-12 15:05:02,192 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:02,192 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:02,192 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. after waiting 0 ms 2023-07-12 15:05:02,192 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:02,192 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:02,192 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1558): Region close journal for ffe3dd29195363fb366cf4b7acdf928f: 2023-07-12 15:05:02,196 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:05:02,197 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174302197"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174302197"}]},"ts":"1689174302197"} 2023-07-12 15:05:02,197 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174302197"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174302197"}]},"ts":"1689174302197"} 2023-07-12 15:05:02,197 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689174302197"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174302197"}]},"ts":"1689174302197"} 2023-07-12 15:05:02,197 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689174302197"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174302197"}]},"ts":"1689174302197"} 2023-07-12 15:05:02,202 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 4 regions to meta. 2023-07-12 15:05:02,203 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:05:02,204 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174302204"}]},"ts":"1689174302204"} 2023-07-12 15:05:02,205 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-12 15:05:02,211 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=365408da311d11f25467a308428c748a, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=d50f055275b35092de8260c021ea4888, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=bad388b7482b4b27d67bc96cd80a265f, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ffe3dd29195363fb366cf4b7acdf928f, ASSIGN}] 2023-07-12 15:05:02,212 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ffe3dd29195363fb366cf4b7acdf928f, ASSIGN 2023-07-12 15:05:02,213 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ffe3dd29195363fb366cf4b7acdf928f, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,35497,1689174298592; forceNewPlan=false, retain=false 2023-07-12 15:05:02,214 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=bad388b7482b4b27d67bc96cd80a265f, ASSIGN 2023-07-12 15:05:02,214 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=bad388b7482b4b27d67bc96cd80a265f, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,35497,1689174298592; forceNewPlan=false, retain=false 2023-07-12 15:05:02,215 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=d50f055275b35092de8260c021ea4888, ASSIGN 2023-07-12 15:05:02,215 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=d50f055275b35092de8260c021ea4888, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,35497,1689174298592; forceNewPlan=false, retain=false 2023-07-12 15:05:02,216 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=365408da311d11f25467a308428c748a, ASSIGN 2023-07-12 15:05:02,216 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=365408da311d11f25467a308428c748a, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,35497,1689174298592; forceNewPlan=false, retain=false 2023-07-12 15:05:02,366 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=bad388b7482b4b27d67bc96cd80a265f, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:02,366 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174302366"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174302366"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174302366"}]},"ts":"1689174302366"} 2023-07-12 15:05:02,369 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=12, state=RUNNABLE; OpenRegionProcedure bad388b7482b4b27d67bc96cd80a265f, server=jenkins-hbase5.apache.org,35497,1689174298592}] 2023-07-12 15:05:02,372 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=ffe3dd29195363fb366cf4b7acdf928f, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:02,372 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689174302372"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174302372"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174302372"}]},"ts":"1689174302372"} 2023-07-12 15:05:02,375 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=13, state=RUNNABLE; OpenRegionProcedure ffe3dd29195363fb366cf4b7acdf928f, server=jenkins-hbase5.apache.org,35497,1689174298592}] 2023-07-12 15:05:02,378 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=d50f055275b35092de8260c021ea4888, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:02,378 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174302378"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174302378"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174302378"}]},"ts":"1689174302378"} 2023-07-12 15:05:02,380 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=11, state=RUNNABLE; OpenRegionProcedure d50f055275b35092de8260c021ea4888, server=jenkins-hbase5.apache.org,35497,1689174298592}] 2023-07-12 15:05:02,385 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=365408da311d11f25467a308428c748a, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:02,385 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689174302385"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174302385"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174302385"}]},"ts":"1689174302385"} 2023-07-12 15:05:02,387 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=10, state=RUNNABLE; OpenRegionProcedure 365408da311d11f25467a308428c748a, server=jenkins-hbase5.apache.org,35497,1689174298592}] 2023-07-12 15:05:02,526 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:02,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d50f055275b35092de8260c021ea4888, NAME => 'testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888.', STARTKEY => 'a', ENDKEY => 'b'} 2023-07-12 15:05:02,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 d50f055275b35092de8260c021ea4888 2023-07-12 15:05:02,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:02,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for d50f055275b35092de8260c021ea4888 2023-07-12 15:05:02,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for d50f055275b35092de8260c021ea4888 2023-07-12 15:05:02,529 INFO [StoreOpener-d50f055275b35092de8260c021ea4888-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region d50f055275b35092de8260c021ea4888 2023-07-12 15:05:02,531 DEBUG [StoreOpener-d50f055275b35092de8260c021ea4888-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888/f1 2023-07-12 15:05:02,531 DEBUG [StoreOpener-d50f055275b35092de8260c021ea4888-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888/f1 2023-07-12 15:05:02,532 INFO [StoreOpener-d50f055275b35092de8260c021ea4888-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d50f055275b35092de8260c021ea4888 columnFamilyName f1 2023-07-12 15:05:02,532 INFO [StoreOpener-d50f055275b35092de8260c021ea4888-1] regionserver.HStore(310): Store=d50f055275b35092de8260c021ea4888/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:02,532 INFO [StoreOpener-d50f055275b35092de8260c021ea4888-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region d50f055275b35092de8260c021ea4888 2023-07-12 15:05:02,534 DEBUG [StoreOpener-d50f055275b35092de8260c021ea4888-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888/f2 2023-07-12 15:05:02,534 DEBUG [StoreOpener-d50f055275b35092de8260c021ea4888-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888/f2 2023-07-12 15:05:02,534 INFO [StoreOpener-d50f055275b35092de8260c021ea4888-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d50f055275b35092de8260c021ea4888 columnFamilyName f2 2023-07-12 15:05:02,534 INFO [StoreOpener-d50f055275b35092de8260c021ea4888-1] regionserver.HStore(310): Store=d50f055275b35092de8260c021ea4888/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:02,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888 2023-07-12 15:05:02,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888 2023-07-12 15:05:02,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:05:02,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for d50f055275b35092de8260c021ea4888 2023-07-12 15:05:02,547 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:05:02,548 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened d50f055275b35092de8260c021ea4888; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71311357, jitterRate=0.06262202560901642}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:05:02,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for d50f055275b35092de8260c021ea4888: 2023-07-12 15:05:02,550 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888., pid=16, masterSystemTime=1689174302521 2023-07-12 15:05:02,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:02,552 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:02,552 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:02,552 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=d50f055275b35092de8260c021ea4888, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:02,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ffe3dd29195363fb366cf4b7acdf928f, NAME => 'testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f.', STARTKEY => 'c', ENDKEY => ''} 2023-07-12 15:05:02,552 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174302552"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174302552"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174302552"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174302552"}]},"ts":"1689174302552"} 2023-07-12 15:05:02,553 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:02,553 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:02,553 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:02,553 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:02,556 INFO [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:02,557 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=11 2023-07-12 15:05:02,557 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=11, state=SUCCESS; OpenRegionProcedure d50f055275b35092de8260c021ea4888, server=jenkins-hbase5.apache.org,35497,1689174298592 in 175 msec 2023-07-12 15:05:02,558 DEBUG [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f/f1 2023-07-12 15:05:02,558 DEBUG [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f/f1 2023-07-12 15:05:02,558 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=d50f055275b35092de8260c021ea4888, ASSIGN in 346 msec 2023-07-12 15:05:02,559 INFO [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ffe3dd29195363fb366cf4b7acdf928f columnFamilyName f1 2023-07-12 15:05:02,559 INFO [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] regionserver.HStore(310): Store=ffe3dd29195363fb366cf4b7acdf928f/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:02,560 INFO [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:02,561 DEBUG [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f/f2 2023-07-12 15:05:02,561 DEBUG [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f/f2 2023-07-12 15:05:02,562 INFO [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ffe3dd29195363fb366cf4b7acdf928f columnFamilyName f2 2023-07-12 15:05:02,562 INFO [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] regionserver.HStore(310): Store=ffe3dd29195363fb366cf4b7acdf928f/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:02,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:02,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:02,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:05:02,569 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:02,575 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:05:02,576 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened ffe3dd29195363fb366cf4b7acdf928f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61644760, jitterRate=-0.08142149448394775}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:05:02,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for ffe3dd29195363fb366cf4b7acdf928f: 2023-07-12 15:05:02,577 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f., pid=15, masterSystemTime=1689174302521 2023-07-12 15:05:02,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:02,578 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:02,578 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:02,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 365408da311d11f25467a308428c748a, NAME => 'testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a.', STARTKEY => '', ENDKEY => 'a'} 2023-07-12 15:05:02,579 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=ffe3dd29195363fb366cf4b7acdf928f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:02,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 365408da311d11f25467a308428c748a 2023-07-12 15:05:02,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:02,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 365408da311d11f25467a308428c748a 2023-07-12 15:05:02,579 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689174302579"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174302579"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174302579"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174302579"}]},"ts":"1689174302579"} 2023-07-12 15:05:02,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 365408da311d11f25467a308428c748a 2023-07-12 15:05:02,580 INFO [StoreOpener-365408da311d11f25467a308428c748a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 365408da311d11f25467a308428c748a 2023-07-12 15:05:02,582 DEBUG [StoreOpener-365408da311d11f25467a308428c748a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a/f1 2023-07-12 15:05:02,582 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=13 2023-07-12 15:05:02,582 DEBUG [StoreOpener-365408da311d11f25467a308428c748a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a/f1 2023-07-12 15:05:02,582 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=13, state=SUCCESS; OpenRegionProcedure ffe3dd29195363fb366cf4b7acdf928f, server=jenkins-hbase5.apache.org,35497,1689174298592 in 205 msec 2023-07-12 15:05:02,583 INFO [StoreOpener-365408da311d11f25467a308428c748a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 365408da311d11f25467a308428c748a columnFamilyName f1 2023-07-12 15:05:02,583 INFO [StoreOpener-365408da311d11f25467a308428c748a-1] regionserver.HStore(310): Store=365408da311d11f25467a308428c748a/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:02,583 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ffe3dd29195363fb366cf4b7acdf928f, ASSIGN in 371 msec 2023-07-12 15:05:02,584 INFO [StoreOpener-365408da311d11f25467a308428c748a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 365408da311d11f25467a308428c748a 2023-07-12 15:05:02,585 DEBUG [StoreOpener-365408da311d11f25467a308428c748a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a/f2 2023-07-12 15:05:02,585 DEBUG [StoreOpener-365408da311d11f25467a308428c748a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a/f2 2023-07-12 15:05:02,585 INFO [StoreOpener-365408da311d11f25467a308428c748a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 365408da311d11f25467a308428c748a columnFamilyName f2 2023-07-12 15:05:02,586 INFO [StoreOpener-365408da311d11f25467a308428c748a-1] regionserver.HStore(310): Store=365408da311d11f25467a308428c748a/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:02,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a 2023-07-12 15:05:02,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a 2023-07-12 15:05:02,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:05:02,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 365408da311d11f25467a308428c748a 2023-07-12 15:05:02,605 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:05:02,606 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 365408da311d11f25467a308428c748a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68226927, jitterRate=0.016660436987876892}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:05:02,606 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 365408da311d11f25467a308428c748a: 2023-07-12 15:05:02,607 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a., pid=17, masterSystemTime=1689174302521 2023-07-12 15:05:02,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:02,609 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:02,610 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:02,610 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=365408da311d11f25467a308428c748a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:02,610 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => bad388b7482b4b27d67bc96cd80a265f, NAME => 'testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f.', STARTKEY => 'b', ENDKEY => 'c'} 2023-07-12 15:05:02,610 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689174302610"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174302610"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174302610"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174302610"}]},"ts":"1689174302610"} 2023-07-12 15:05:02,610 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:02,610 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:02,610 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:02,610 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:02,613 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=10 2023-07-12 15:05:02,614 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=10, state=SUCCESS; OpenRegionProcedure 365408da311d11f25467a308428c748a, server=jenkins-hbase5.apache.org,35497,1689174298592 in 225 msec 2023-07-12 15:05:02,615 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=365408da311d11f25467a308428c748a, ASSIGN in 403 msec 2023-07-12 15:05:02,623 INFO [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:02,625 DEBUG [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f/f1 2023-07-12 15:05:02,626 DEBUG [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f/f1 2023-07-12 15:05:02,626 INFO [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bad388b7482b4b27d67bc96cd80a265f columnFamilyName f1 2023-07-12 15:05:02,627 INFO [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] regionserver.HStore(310): Store=bad388b7482b4b27d67bc96cd80a265f/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:02,627 INFO [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:02,628 DEBUG [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f/f2 2023-07-12 15:05:02,628 DEBUG [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f/f2 2023-07-12 15:05:02,629 INFO [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bad388b7482b4b27d67bc96cd80a265f columnFamilyName f2 2023-07-12 15:05:02,629 INFO [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] regionserver.HStore(310): Store=bad388b7482b4b27d67bc96cd80a265f/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:02,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:02,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:02,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:05:02,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:02,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:05:02,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened bad388b7482b4b27d67bc96cd80a265f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68991081, jitterRate=0.028047218918800354}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:05:02,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for bad388b7482b4b27d67bc96cd80a265f: 2023-07-12 15:05:02,636 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f., pid=14, masterSystemTime=1689174302521 2023-07-12 15:05:02,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:02,638 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:02,638 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=bad388b7482b4b27d67bc96cd80a265f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:02,638 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174302638"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174302638"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174302638"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174302638"}]},"ts":"1689174302638"} 2023-07-12 15:05:02,642 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=12 2023-07-12 15:05:02,642 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=12, state=SUCCESS; OpenRegionProcedure bad388b7482b4b27d67bc96cd80a265f, server=jenkins-hbase5.apache.org,35497,1689174298592 in 271 msec 2023-07-12 15:05:02,644 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=12, resume processing ppid=9 2023-07-12 15:05:02,644 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=bad388b7482b4b27d67bc96cd80a265f, ASSIGN in 431 msec 2023-07-12 15:05:02,644 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 15:05:02,645 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174302645"}]},"ts":"1689174302645"} 2023-07-12 15:05:02,646 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=ENABLED in hbase:meta 2023-07-12 15:05:02,656 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 15:05:02,657 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 in 1.3320 sec 2023-07-12 15:05:02,828 DEBUG [Listener at localhost/44677] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 15:05:02,831 INFO [RS-EventLoopGroup-15-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:50658, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 15:05:02,831 INFO [Listener at localhost/44677] client.HBaseAdmin$15(890): Started disable of testEnableTableWithFailoverAtStep3 2023-07-12 15:05:02,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34909] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testEnableTableWithFailoverAtStep3 2023-07-12 15:05:02,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34909] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:02,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34909] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 15:05:02,836 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174302836"}]},"ts":"1689174302836"} 2023-07-12 15:05:02,837 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=DISABLING in hbase:meta 2023-07-12 15:05:02,839 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testEnableTableWithFailoverAtStep3 to state=DISABLING 2023-07-12 15:05:02,839 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=365408da311d11f25467a308428c748a, UNASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=d50f055275b35092de8260c021ea4888, UNASSIGN}, {pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=bad388b7482b4b27d67bc96cd80a265f, UNASSIGN}, {pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ffe3dd29195363fb366cf4b7acdf928f, UNASSIGN}] 2023-07-12 15:05:02,840 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ffe3dd29195363fb366cf4b7acdf928f, UNASSIGN 2023-07-12 15:05:02,841 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=ffe3dd29195363fb366cf4b7acdf928f, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:02,841 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689174302841"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174302841"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174302841"}]},"ts":"1689174302841"} 2023-07-12 15:05:02,842 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; CloseRegionProcedure ffe3dd29195363fb366cf4b7acdf928f, server=jenkins-hbase5.apache.org,35497,1689174298592}] 2023-07-12 15:05:02,844 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=bad388b7482b4b27d67bc96cd80a265f, UNASSIGN 2023-07-12 15:05:02,844 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=bad388b7482b4b27d67bc96cd80a265f, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:02,845 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174302844"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174302844"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174302844"}]},"ts":"1689174302844"} 2023-07-12 15:05:02,846 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=21, state=RUNNABLE; CloseRegionProcedure bad388b7482b4b27d67bc96cd80a265f, server=jenkins-hbase5.apache.org,35497,1689174298592}] 2023-07-12 15:05:02,848 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=d50f055275b35092de8260c021ea4888, UNASSIGN 2023-07-12 15:05:02,848 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=d50f055275b35092de8260c021ea4888, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:02,848 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174302848"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174302848"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174302848"}]},"ts":"1689174302848"} 2023-07-12 15:05:02,850 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=20, state=RUNNABLE; CloseRegionProcedure d50f055275b35092de8260c021ea4888, server=jenkins-hbase5.apache.org,35497,1689174298592}] 2023-07-12 15:05:02,851 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=365408da311d11f25467a308428c748a, UNASSIGN 2023-07-12 15:05:02,852 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=365408da311d11f25467a308428c748a, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:02,852 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689174302852"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174302852"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174302852"}]},"ts":"1689174302852"} 2023-07-12 15:05:02,853 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=19, state=RUNNABLE; CloseRegionProcedure 365408da311d11f25467a308428c748a, server=jenkins-hbase5.apache.org,35497,1689174298592}] 2023-07-12 15:05:02,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34909] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 15:05:02,994 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:02,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing bad388b7482b4b27d67bc96cd80a265f, disabling compactions & flushes 2023-07-12 15:05:02,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:02,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:02,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. after waiting 0 ms 2023-07-12 15:05:02,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:02,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 15:05:03,000 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:03,000 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for bad388b7482b4b27d67bc96cd80a265f: 2023-07-12 15:05:03,001 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:03,001 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 365408da311d11f25467a308428c748a 2023-07-12 15:05:03,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 365408da311d11f25467a308428c748a, disabling compactions & flushes 2023-07-12 15:05:03,002 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:03,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:03,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. after waiting 0 ms 2023-07-12 15:05:03,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:03,002 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=bad388b7482b4b27d67bc96cd80a265f, regionState=CLOSED 2023-07-12 15:05:03,003 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174303002"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174303002"}]},"ts":"1689174303002"} 2023-07-12 15:05:03,006 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=21 2023-07-12 15:05:03,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 15:05:03,006 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=21, state=SUCCESS; CloseRegionProcedure bad388b7482b4b27d67bc96cd80a265f, server=jenkins-hbase5.apache.org,35497,1689174298592 in 158 msec 2023-07-12 15:05:03,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:03,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 365408da311d11f25467a308428c748a: 2023-07-12 15:05:03,007 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=bad388b7482b4b27d67bc96cd80a265f, UNASSIGN in 167 msec 2023-07-12 15:05:03,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 365408da311d11f25467a308428c748a 2023-07-12 15:05:03,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close d50f055275b35092de8260c021ea4888 2023-07-12 15:05:03,008 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing d50f055275b35092de8260c021ea4888, disabling compactions & flushes 2023-07-12 15:05:03,009 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:03,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:03,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. after waiting 0 ms 2023-07-12 15:05:03,009 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=365408da311d11f25467a308428c748a, regionState=CLOSED 2023-07-12 15:05:03,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:03,009 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689174303009"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174303009"}]},"ts":"1689174303009"} 2023-07-12 15:05:03,012 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=19 2023-07-12 15:05:03,012 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=19, state=SUCCESS; CloseRegionProcedure 365408da311d11f25467a308428c748a, server=jenkins-hbase5.apache.org,35497,1689174298592 in 157 msec 2023-07-12 15:05:03,013 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 15:05:03,013 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=365408da311d11f25467a308428c748a, UNASSIGN in 173 msec 2023-07-12 15:05:03,013 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:03,013 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for d50f055275b35092de8260c021ea4888: 2023-07-12 15:05:03,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed d50f055275b35092de8260c021ea4888 2023-07-12 15:05:03,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:03,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing ffe3dd29195363fb366cf4b7acdf928f, disabling compactions & flushes 2023-07-12 15:05:03,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:03,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:03,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. after waiting 0 ms 2023-07-12 15:05:03,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:03,015 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=d50f055275b35092de8260c021ea4888, regionState=CLOSED 2023-07-12 15:05:03,016 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174303015"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174303015"}]},"ts":"1689174303015"} 2023-07-12 15:05:03,018 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=20 2023-07-12 15:05:03,018 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=20, state=SUCCESS; CloseRegionProcedure d50f055275b35092de8260c021ea4888, server=jenkins-hbase5.apache.org,35497,1689174298592 in 167 msec 2023-07-12 15:05:03,019 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 15:05:03,019 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=d50f055275b35092de8260c021ea4888, UNASSIGN in 179 msec 2023-07-12 15:05:03,019 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:03,019 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for ffe3dd29195363fb366cf4b7acdf928f: 2023-07-12 15:05:03,020 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:03,021 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=ffe3dd29195363fb366cf4b7acdf928f, regionState=CLOSED 2023-07-12 15:05:03,021 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689174303021"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689174303021"}]},"ts":"1689174303021"} 2023-07-12 15:05:03,023 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-12 15:05:03,023 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; CloseRegionProcedure ffe3dd29195363fb366cf4b7acdf928f, server=jenkins-hbase5.apache.org,35497,1689174298592 in 180 msec 2023-07-12 15:05:03,024 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=18 2023-07-12 15:05:03,024 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ffe3dd29195363fb366cf4b7acdf928f, UNASSIGN in 184 msec 2023-07-12 15:05:03,025 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174303025"}]},"ts":"1689174303025"} 2023-07-12 15:05:03,026 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=DISABLED in hbase:meta 2023-07-12 15:05:03,028 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testEnableTableWithFailoverAtStep3 to state=DISABLED 2023-07-12 15:05:03,030 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 in 197 msec 2023-07-12 15:05:03,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34909] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 15:05:03,138 INFO [Listener at localhost/44677] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testEnableTableWithFailoverAtStep3, procId: 18 completed 2023-07-12 15:05:03,139 WARN [Listener at localhost/44677] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-07-12 15:05:03,144 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,146 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:05:03,146 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,146 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:05:03,146 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:05:03,289 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-12 15:05:03,394 INFO [Listener at localhost/44677] procedure.TestMasterFailoverWithProcedures(306): Restart 0 exec state: pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,394 INFO [Listener at localhost/44677] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:05:03,394 INFO [Listener at localhost/44677] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:05:03,394 INFO [Listener at localhost/44677] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:05:03,396 ERROR [Listener at localhost/44677] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:05:03,396 INFO [Listener at localhost/44677] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:05:03,396 INFO [Listener at localhost/44677] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:05:03,396 INFO [Listener at localhost/44677] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:05:03,396 INFO [Listener at localhost/44677] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:05:03,397 INFO [Listener at localhost/44677] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:05:03,398 DEBUG [Listener at localhost/44677] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.EnableTableProcedure) stack ids=[] 2023-07-12 15:05:03,398 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:05:03,399 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:05:03,399 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:05:03,399 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:05:03,399 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,399 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,400 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,400 DEBUG [Listener at localhost/44677] procedure2.Procedure(966): pid=27, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:05:03,400 INFO [Listener at localhost/44677] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 15:05:03,401 DEBUG [Listener at localhost/44677] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:05:03,401 DEBUG [Listener at localhost/44677] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,35497,1689174298592, table=hbase:meta, region=1588230740 2023-07-12 15:05:03,401 INFO [Listener at localhost/44677] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174333401 2023-07-12 15:05:03,402 DEBUG [Listener at localhost/44677] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:05:03,402 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,404 DEBUG [Listener at localhost/44677] assignment.RegionStateStore(147): Load hbase:meta entry region=b4ef2da9a8e7a43cdb82478409694269, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:03,405 DEBUG [Listener at localhost/44677] assignment.RegionStateStore(147): Load hbase:meta entry region=365408da311d11f25467a308428c748a, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:03,405 DEBUG [Listener at localhost/44677] assignment.RegionStateStore(147): Load hbase:meta entry region=d50f055275b35092de8260c021ea4888, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:03,405 DEBUG [Listener at localhost/44677] assignment.RegionStateStore(147): Load hbase:meta entry region=bad388b7482b4b27d67bc96cd80a265f, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:03,405 DEBUG [Listener at localhost/44677] assignment.RegionStateStore(147): Load hbase:meta entry region=ffe3dd29195363fb366cf4b7acdf928f, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:03,405 INFO [Listener at localhost/44677] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:05:03,405 INFO [Listener at localhost/44677] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174363405 2023-07-12 15:05:03,405 INFO [Listener at localhost/44677] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174423405 2023-07-12 15:05:03,405 INFO [Listener at localhost/44677] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 15:05:03,405 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:05:03,407 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:05:03,407 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_SET_ENABLING_TABLE_STATE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,407 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:05:03,407 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_SET_ENABLING_TABLE_STATE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:05:03,655 INFO [Listener at localhost/44677] procedure.TestMasterFailoverWithProcedures(306): Restart 1 exec state: pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,656 INFO [Listener at localhost/44677] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:05:03,656 INFO [Listener at localhost/44677] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:05:03,656 INFO [Listener at localhost/44677] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:05:03,656 ERROR [Listener at localhost/44677] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:05:03,656 INFO [Listener at localhost/44677] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:05:03,656 INFO [Listener at localhost/44677] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:05:03,656 INFO [Listener at localhost/44677] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:05:03,656 INFO [Listener at localhost/44677] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:05:03,657 INFO [Listener at localhost/44677] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:05:03,658 DEBUG [Listener at localhost/44677] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.EnableTableProcedure) stack ids=[0] 2023-07-12 15:05:03,658 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:05:03,658 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:05:03,658 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:05:03,659 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:05:03,659 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,659 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,660 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,660 DEBUG [Listener at localhost/44677] procedure2.Procedure(966): pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:05:03,660 INFO [Listener at localhost/44677] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 15:05:03,661 DEBUG [Listener at localhost/44677] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:05:03,661 DEBUG [Listener at localhost/44677] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,35497,1689174298592, table=hbase:meta, region=1588230740 2023-07-12 15:05:03,663 INFO [Listener at localhost/44677] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174333663 2023-07-12 15:05:03,666 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,666 DEBUG [Listener at localhost/44677] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:05:03,668 DEBUG [Listener at localhost/44677] assignment.RegionStateStore(147): Load hbase:meta entry region=b4ef2da9a8e7a43cdb82478409694269, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:03,669 DEBUG [Listener at localhost/44677] assignment.RegionStateStore(147): Load hbase:meta entry region=365408da311d11f25467a308428c748a, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:03,669 DEBUG [Listener at localhost/44677] assignment.RegionStateStore(147): Load hbase:meta entry region=d50f055275b35092de8260c021ea4888, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:03,669 DEBUG [Listener at localhost/44677] assignment.RegionStateStore(147): Load hbase:meta entry region=bad388b7482b4b27d67bc96cd80a265f, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:03,669 DEBUG [Listener at localhost/44677] assignment.RegionStateStore(147): Load hbase:meta entry region=ffe3dd29195363fb366cf4b7acdf928f, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:03,669 INFO [Listener at localhost/44677] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:05:03,669 INFO [Listener at localhost/44677] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174363669 2023-07-12 15:05:03,670 INFO [Listener at localhost/44677] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174423670 2023-07-12 15:05:03,670 INFO [Listener at localhost/44677] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 15:05:03,670 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:05:03,672 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,672 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174303672"}]},"ts":"1689174303672"} 2023-07-12 15:05:03,674 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-12 15:05:03,677 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:05:03,677 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_MARK_REGIONS_ONLINE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,677 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:05:03,677 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_MARK_REGIONS_ONLINE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:05:03,920 INFO [Listener at localhost/44677] procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state: pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,920 INFO [Listener at localhost/44677] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 15:05:03,920 INFO [Listener at localhost/44677] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:05:03,920 INFO [Listener at localhost/44677] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:05:03,920 ERROR [Listener at localhost/44677] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:05:03,920 INFO [Listener at localhost/44677] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 15:05:03,920 INFO [Listener at localhost/44677] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:05:03,921 INFO [Listener at localhost/44677] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:05:03,921 INFO [Listener at localhost/44677] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:05:03,921 INFO [Listener at localhost/44677] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:05:03,922 DEBUG [Listener at localhost/44677] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.EnableTableProcedure) stack ids=[0, 1] 2023-07-12 15:05:03,922 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:05:03,923 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:05:03,923 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:05:03,923 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:05:03,923 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,924 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,924 DEBUG [Listener at localhost/44677] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:ENABLE_TABLE_SET_ENABLING_TABLE_STATE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,924 DEBUG [Listener at localhost/44677] procedure2.Procedure(966): pid=27, state=RUNNABLE:ENABLE_TABLE_SET_ENABLING_TABLE_STATE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:05:03,924 INFO [Listener at localhost/44677] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 15:05:03,925 DEBUG [Listener at localhost/44677] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:05:03,925 DEBUG [Listener at localhost/44677] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,35497,1689174298592, table=hbase:meta, region=1588230740 2023-07-12 15:05:03,925 INFO [Listener at localhost/44677] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174333925 2023-07-12 15:05:03,926 DEBUG [Listener at localhost/44677] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:05:03,926 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:ENABLE_TABLE_SET_ENABLING_TABLE_STATE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,929 DEBUG [Listener at localhost/44677] assignment.RegionStateStore(147): Load hbase:meta entry region=b4ef2da9a8e7a43cdb82478409694269, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:03,929 DEBUG [Listener at localhost/44677] assignment.RegionStateStore(147): Load hbase:meta entry region=365408da311d11f25467a308428c748a, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:03,930 DEBUG [Listener at localhost/44677] assignment.RegionStateStore(147): Load hbase:meta entry region=d50f055275b35092de8260c021ea4888, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:03,930 DEBUG [Listener at localhost/44677] assignment.RegionStateStore(147): Load hbase:meta entry region=bad388b7482b4b27d67bc96cd80a265f, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:03,930 DEBUG [Listener at localhost/44677] assignment.RegionStateStore(147): Load hbase:meta entry region=ffe3dd29195363fb366cf4b7acdf928f, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:03,930 INFO [Listener at localhost/44677] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:05:03,930 INFO [Listener at localhost/44677] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174363930 2023-07-12 15:05:03,930 INFO [Listener at localhost/44677] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174423930 2023-07-12 15:05:03,930 INFO [Listener at localhost/44677] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-07-12 15:05:03,931 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,931 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174303931"}]},"ts":"1689174303931"} 2023-07-12 15:05:03,932 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-12 15:05:03,934 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 15:05:03,936 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-07-12 15:05:03,936 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=365408da311d11f25467a308428c748a, ASSIGN}, {pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=d50f055275b35092de8260c021ea4888, ASSIGN}, {pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=bad388b7482b4b27d67bc96cd80a265f, ASSIGN}, {pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ffe3dd29195363fb366cf4b7acdf928f, ASSIGN}] 2023-07-12 15:05:03,936 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 15:05:03,936 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=WAITING:ENABLE_TABLE_SET_ENABLED_TABLE_STATE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:03,936 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:05:03,936 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=WAITING:ENABLE_TABLE_SET_ENABLED_TABLE_STATE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 15:05:04,181 INFO [Listener at localhost/44677] procedure.TestMasterFailoverWithProcedures(313): Trigger master failover 2023-07-12 15:05:04,181 INFO [Listener at localhost/44677] hbase.MiniHBaseCluster(607): Aborting jenkins-hbase5.apache.org,34909,1689174298540 2023-07-12 15:05:04,181 ERROR [Listener at localhost/44677] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 15:05:04,181 ERROR [Listener at localhost/44677] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase5.apache.org,34909,1689174298540: Aborting for tests ***** java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:608) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:411) at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.masterFailover(MasterProcedureTestingUtility.java:135) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testRecoveryAndDoubleExecution(TestMasterFailoverWithProcedures.java:314) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testEnableTableWithFailoverAtStep(TestMasterFailoverWithProcedures.java:289) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testEnableTableWithFailover(TestMasterFailoverWithProcedures.java:270) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:05:04,181 INFO [Listener at localhost/44677] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,34909,1689174298540' ***** 2023-07-12 15:05:04,181 INFO [Listener at localhost/44677] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost/44677 2023-07-12 15:05:04,182 DEBUG [M:0;jenkins-hbase5:34909] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6543b831, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:05:04,182 INFO [M:0;jenkins-hbase5:34909] regionserver.HRegionServer(1141): aborting server jenkins-hbase5.apache.org,34909,1689174298540 2023-07-12 15:05:04,182 DEBUG [M:0;jenkins-hbase5:34909] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e1e3eaa to 127.0.0.1:51235 2023-07-12 15:05:04,183 DEBUG [M:0;jenkins-hbase5:34909] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:05:04,183 INFO [M:0;jenkins-hbase5:34909] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,34909,1689174298540; all regions closed. 2023-07-12 15:05:04,183 DEBUG [M:0;jenkins-hbase5:34909] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:05:04,183 DEBUG [M:0;jenkins-hbase5:34909] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 15:05:04,183 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 15:05:04,183 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174299548] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174299548,5,FailOnTimeoutGroup] 2023-07-12 15:05:04,183 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174299548] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174299548,5,FailOnTimeoutGroup] 2023-07-12 15:05:04,183 DEBUG [M:0;jenkins-hbase5:34909] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 15:05:04,185 INFO [M:0;jenkins-hbase5:34909] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 15:05:04,185 INFO [M:0;jenkins-hbase5:34909] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 15:05:04,185 INFO [M:0;jenkins-hbase5:34909] hbase.ChoreService(369): Chore service for: master/jenkins-hbase5:0 had [] on shutdown 2023-07-12 15:05:04,185 DEBUG [M:0;jenkins-hbase5:34909] master.HMaster(1512): Stopping service threads 2023-07-12 15:05:04,185 INFO [M:0;jenkins-hbase5:34909] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 15:05:04,185 ERROR [M:0;jenkins-hbase5:34909] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:05:04,185 INFO [M:0;jenkins-hbase5:34909] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 15:05:04,185 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 15:05:04,185 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:05:04,186 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:05:04,186 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:05:04,186 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:05:04,186 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): regionserver:35497-0x101c5ad18690002, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:05:04,186 DEBUG [M:0;jenkins-hbase5:34909] zookeeper.ZKUtil(398): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 15:05:04,186 WARN [M:0;jenkins-hbase5:34909] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 15:05:04,186 INFO [M:0;jenkins-hbase5:34909] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:05:04,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:05:04,186 INFO [M:0;jenkins-hbase5:34909] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 15:05:04,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:05:04,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35497-0x101c5ad18690002, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:05:04,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:05:04,187 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-12 15:05:04,188 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:05:04,188 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): regionserver:35497-0x101c5ad18690002, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:05:04,188 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:05:04,188 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:05:04,189 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:05:04,189 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:05:04,189 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase5.apache.org,46407,1689174298568 from backup master directory 2023-07-12 15:05:04,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:05:04,190 DEBUG [M:0;jenkins-hbase5:34909] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:05:04,190 INFO [M:0;jenkins-hbase5:34909] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:05:04,190 DEBUG [M:0;jenkins-hbase5:34909] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:05:04,190 DEBUG [M:0;jenkins-hbase5:34909] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:05:04,190 DEBUG [M:0;jenkins-hbase5:34909] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:05:04,191 ERROR [M:0;jenkins-hbase5:34909] regionserver.HRegion(1825): Memstore data size is 89379 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:05:04,191 INFO [M:0;jenkins-hbase5:34909] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:05:04,191 DEBUG [M:0;jenkins-hbase5:34909] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:05:04,191 INFO [M:0;jenkins-hbase5:34909] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 15:05:04,191 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:05:04,191 INFO [M:0;jenkins-hbase5:34909] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:34909 2023-07-12 15:05:04,192 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase5.apache.org,46407,1689174298568 2023-07-12 15:05:04,192 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:05:04,192 WARN [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:05:04,192 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:05:04,192 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase5.apache.org,46407,1689174298568 2023-07-12 15:05:04,193 DEBUG [M:0;jenkins-hbase5:34909] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase5.apache.org,34909,1689174298540 already deleted, retry=false 2023-07-12 15:05:04,205 INFO [master/jenkins-hbase5:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:05:04,221 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x1a94c521 to 127.0.0.1:51235 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:05:04,225 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1686d1cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:05:04,226 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 15:05:04,226 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 15:05:04,226 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:05:04,231 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/WALs/jenkins-hbase5.apache.org,34909,1689174298540 to hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/WALs/jenkins-hbase5.apache.org,34909,1689174298540-dead as it is dead 2023-07-12 15:05:04,232 INFO [master/jenkins-hbase5:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/WALs/jenkins-hbase5.apache.org,34909,1689174298540-dead/jenkins-hbase5.apache.org%2C34909%2C1689174298540.1689174299477 2023-07-12 15:05:04,232 INFO [master/jenkins-hbase5:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/WALs/jenkins-hbase5.apache.org,34909,1689174298540-dead/jenkins-hbase5.apache.org%2C34909%2C1689174298540.1689174299477 after 0ms 2023-07-12 15:05:04,233 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/WALs/jenkins-hbase5.apache.org,34909,1689174298540-dead/jenkins-hbase5.apache.org%2C34909%2C1689174298540.1689174299477 to hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C34909%2C1689174298540.1689174299477 2023-07-12 15:05:04,233 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/WALs/jenkins-hbase5.apache.org,34909,1689174298540-dead 2023-07-12 15:05:04,233 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/WALs/jenkins-hbase5.apache.org,46407,1689174298568 2023-07-12 15:05:04,235 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C46407%2C1689174298568, suffix=, logDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/WALs/jenkins-hbase5.apache.org,46407,1689174298568, archiveDir=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/oldWALs, maxLogs=10 2023-07-12 15:05:04,246 WARN [RS-EventLoopGroup-17-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase5.apache.org/172.31.10.131:34909 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:34909 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:05:04,247 DEBUG [RS-EventLoopGroup-17-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase5.apache.org/172.31.10.131:34909 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:34909 2023-07-12 15:05:04,254 DEBUG [RS-EventLoopGroup-17-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36041,DS-8861a867-3cf6-47c1-b2f9-7ab4a25576e7,DISK] 2023-07-12 15:05:04,256 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/WALs/jenkins-hbase5.apache.org,46407,1689174298568/jenkins-hbase5.apache.org%2C46407%2C1689174298568.1689174304235 2023-07-12 15:05:04,256 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36041,DS-8861a867-3cf6-47c1-b2f9-7ab4a25576e7,DISK]] 2023-07-12 15:05:04,256 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:05:04,256 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:04,257 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:05:04,257 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:05:04,267 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:05:04,269 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 15:05:04,269 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 15:05:04,269 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:04,270 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-12 15:05:04,270 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C34909%2C1689174298540.1689174299477 2023-07-12 15:05:04,277 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 231, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=193, path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C34909%2C1689174298540.1689174299477 2023-07-12 15:05:04,277 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=87.28 KB heapSize=104.57 KB 2023-07-12 15:05:04,348 DEBUG [RS:0;jenkins-hbase5:35497] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 15:05:04,355 INFO [RS-EventLoopGroup-16-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:53309, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 15:05:04,356 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46407] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:05:04,386 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:05:04,386 INFO [M:0;jenkins-hbase5:34909] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,34909,1689174298540; zookeeper connection closed. 2023-07-12 15:05:04,386 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:34909-0x101c5ad18690000, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:05:04,460 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46407] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:05:04,562 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46407] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:05:04,664 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46407] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:05:04,685 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=87.28 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/22f11be19b004ca5b8333434ed8f1f3d 2023-07-12 15:05:04,691 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/22f11be19b004ca5b8333434ed8f1f3d as hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/22f11be19b004ca5b8333434ed8f1f3d 2023-07-12 15:05:04,696 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/22f11be19b004ca5b8333434ed8f1f3d, entries=231, sequenceid=193, filesize=92.5 K 2023-07-12 15:05:04,696 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~87.28 KB/89379, heapSize ~104.55 KB/107064, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 419ms, sequenceid=193, compaction requested=false; wal=null 2023-07-12 15:05:04,697 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase5.apache.org%2C34909%2C1689174298540.1689174299477 2023-07-12 15:05:04,700 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:05:04,702 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/193.seqid, newMaxSeqId=193, maxSeqId=1 2023-07-12 15:05:04,703 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=194; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69725840, jitterRate=0.038995981216430664}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:05:04,703 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:05:04,703 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 15:05:04,704 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 15:05:04,704 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 15:05:04,704 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:05:04,705 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 15:05:04,710 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.EnableTableProcedure) stack ids=[0, 1, 2] 2023-07-12 15:05:04,710 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 15:05:04,710 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 15:05:04,711 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 15:05:04,711 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:05:04,711 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:04,711 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:04,712 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:ENABLE_TABLE_MARK_REGIONS_ONLINE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:04,712 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.Procedure(966): pid=27, state=RUNNABLE:ENABLE_TABLE_MARK_REGIONS_ONLINE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 15:05:04,712 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-07-12 15:05:04,712 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 15:05:04,713 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 15:05:04,713 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase5.apache.org,35497,1689174298592, table=hbase:meta, region=1588230740 2023-07-12 15:05:04,714 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 1 possibly 'live' servers, and 0 'splitting'. 2023-07-12 15:05:04,715 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:04,729 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 15:05:04,729 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 15:05:04,730 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 15:05:04,730 INFO [master/jenkins-hbase5:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 15:05:04,730 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 15:05:04,731 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 15:05:04,731 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 15:05:04,732 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 15:05:04,732 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase5.apache.org,46407,1689174298568, sessionid=0x101c5ad18690001, setting cluster-up flag (Was=true) 2023-07-12 15:05:04,734 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 15:05:04,734 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,46407,1689174298568 2023-07-12 15:05:04,736 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 15:05:04,737 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,46407,1689174298568 2023-07-12 15:05:04,737 WARN [master/jenkins-hbase5:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/.hbase-snapshot/.tmp 2023-07-12 15:05:04,738 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:05:04,738 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:05:04,738 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:05:04,738 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:05:04,738 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase5:0, corePoolSize=10, maxPoolSize=10 2023-07-12 15:05:04,738 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:05:04,738 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:05:04,738 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:05:04,739 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689174334739 2023-07-12 15:05:04,739 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 15:05:04,740 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:ENABLE_TABLE_MARK_REGIONS_ONLINE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 15:05:04,740 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 15:05:04,740 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 15:05:04,740 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 15:05:04,740 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 15:05:04,740 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 15:05:04,740 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:05:04,741 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 15:05:04,741 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 15:05:04,741 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 15:05:04,741 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 15:05:04,741 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 15:05:04,742 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174304741,5,FailOnTimeoutGroup] 2023-07-12 15:05:04,742 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174304742,5,FailOnTimeoutGroup] 2023-07-12 15:05:04,743 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:05:04,743 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 15:05:04,743 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 15:05:04,743 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 15:05:04,743 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689174304743, completionTime=-1 2023-07-12 15:05:04,743 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 15:05:04,743 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:05:04,744 DEBUG [hconnection-0x5cbd2173-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:05:04,745 INFO [RS-EventLoopGroup-17-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:54168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:05:04,747 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=b4ef2da9a8e7a43cdb82478409694269, regionState=OPEN, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:04,747 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=365408da311d11f25467a308428c748a, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:04,747 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=d50f055275b35092de8260c021ea4888, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:04,747 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=bad388b7482b4b27d67bc96cd80a265f, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:04,748 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=ffe3dd29195363fb366cf4b7acdf928f, regionState=CLOSED, lastHost=jenkins-hbase5.apache.org,35497,1689174298592, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592, openSeqNum=2 2023-07-12 15:05:04,748 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:05:04,748 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689174364748 2023-07-12 15:05:04,748 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689174424748 2023-07-12 15:05:04,748 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-07-12 15:05:04,760 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,46407,1689174298568-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:05:04,760 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,46407,1689174298568-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:05:04,760 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,46407,1689174298568-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:05:04,760 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase5:46407, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:05:04,760 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 15:05:04,761 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 15:05:04,763 DEBUG [master/jenkins-hbase5:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 15:05:04,769 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 15:05:04,776 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 15:05:04,776 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 0.584sec 2023-07-12 15:05:04,777 INFO [master/jenkins-hbase5:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 15:05:04,777 INFO [master/jenkins-hbase5:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 15:05:04,777 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-07-12 15:05:04,777 INFO [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 15:05:04,777 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,46407,1689174298568-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 15:05:04,777 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=365408da311d11f25467a308428c748a, ASSIGN}, {pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=d50f055275b35092de8260c021ea4888, ASSIGN}, {pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=bad388b7482b4b27d67bc96cd80a265f, ASSIGN}, {pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ffe3dd29195363fb366cf4b7acdf928f, ASSIGN}] 2023-07-12 15:05:04,777 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,46407,1689174298568-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 15:05:04,780 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ffe3dd29195363fb366cf4b7acdf928f, ASSIGN 2023-07-12 15:05:04,781 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:04,781 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ffe3dd29195363fb366cf4b7acdf928f, ASSIGN; state=CLOSED, location=jenkins-hbase5.apache.org,35497,1689174298592; forceNewPlan=false, retain=true 2023-07-12 15:05:04,782 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=bad388b7482b4b27d67bc96cd80a265f, ASSIGN 2023-07-12 15:05:04,782 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 15:05:04,782 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:04,782 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=bad388b7482b4b27d67bc96cd80a265f, ASSIGN; state=CLOSED, location=jenkins-hbase5.apache.org,35497,1689174298592; forceNewPlan=false, retain=true 2023-07-12 15:05:04,783 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=d50f055275b35092de8260c021ea4888, ASSIGN 2023-07-12 15:05:04,783 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:04,784 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=d50f055275b35092de8260c021ea4888, ASSIGN; state=CLOSED, location=jenkins-hbase5.apache.org,35497,1689174298592; forceNewPlan=false, retain=true 2023-07-12 15:05:04,784 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=365408da311d11f25467a308428c748a, ASSIGN 2023-07-12 15:05:04,785 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:04,785 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=365408da311d11f25467a308428c748a, ASSIGN; state=CLOSED, location=jenkins-hbase5.apache.org,35497,1689174298592; forceNewPlan=false, retain=true 2023-07-12 15:05:04,933 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=bad388b7482b4b27d67bc96cd80a265f, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:04,933 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174304933"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174304933"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174304933"}]},"ts":"1689174304933"} 2023-07-12 15:05:04,935 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=30, state=RUNNABLE; OpenRegionProcedure bad388b7482b4b27d67bc96cd80a265f, server=jenkins-hbase5.apache.org,35497,1689174298592}] 2023-07-12 15:05:04,937 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=ffe3dd29195363fb366cf4b7acdf928f, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:04,937 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689174304937"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174304937"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174304937"}]},"ts":"1689174304937"} 2023-07-12 15:05:04,939 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=31, state=RUNNABLE; OpenRegionProcedure ffe3dd29195363fb366cf4b7acdf928f, server=jenkins-hbase5.apache.org,35497,1689174298592}] 2023-07-12 15:05:04,941 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=d50f055275b35092de8260c021ea4888, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:04,941 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174304941"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174304941"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174304941"}]},"ts":"1689174304941"} 2023-07-12 15:05:04,943 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=29, state=RUNNABLE; OpenRegionProcedure d50f055275b35092de8260c021ea4888, server=jenkins-hbase5.apache.org,35497,1689174298592}] 2023-07-12 15:05:04,945 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=365408da311d11f25467a308428c748a, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:04,945 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689174304944"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689174304944"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689174304944"}]},"ts":"1689174304944"} 2023-07-12 15:05:04,946 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=35, ppid=28, state=RUNNABLE; OpenRegionProcedure 365408da311d11f25467a308428c748a, server=jenkins-hbase5.apache.org,35497,1689174298592}] 2023-07-12 15:05:05,087 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:05,087 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 15:05:05,089 INFO [RS-EventLoopGroup-17-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:54178, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 15:05:05,092 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:05,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 365408da311d11f25467a308428c748a, NAME => 'testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a.', STARTKEY => '', ENDKEY => 'a'} 2023-07-12 15:05:05,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 365408da311d11f25467a308428c748a 2023-07-12 15:05:05,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:05,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 365408da311d11f25467a308428c748a 2023-07-12 15:05:05,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 365408da311d11f25467a308428c748a 2023-07-12 15:05:05,094 INFO [StoreOpener-365408da311d11f25467a308428c748a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 365408da311d11f25467a308428c748a 2023-07-12 15:05:05,096 DEBUG [StoreOpener-365408da311d11f25467a308428c748a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a/f1 2023-07-12 15:05:05,096 DEBUG [StoreOpener-365408da311d11f25467a308428c748a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a/f1 2023-07-12 15:05:05,096 INFO [StoreOpener-365408da311d11f25467a308428c748a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 365408da311d11f25467a308428c748a columnFamilyName f1 2023-07-12 15:05:05,097 INFO [StoreOpener-365408da311d11f25467a308428c748a-1] regionserver.HStore(310): Store=365408da311d11f25467a308428c748a/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:05,097 INFO [StoreOpener-365408da311d11f25467a308428c748a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 365408da311d11f25467a308428c748a 2023-07-12 15:05:05,098 DEBUG [StoreOpener-365408da311d11f25467a308428c748a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a/f2 2023-07-12 15:05:05,098 DEBUG [StoreOpener-365408da311d11f25467a308428c748a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a/f2 2023-07-12 15:05:05,098 INFO [StoreOpener-365408da311d11f25467a308428c748a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 365408da311d11f25467a308428c748a columnFamilyName f2 2023-07-12 15:05:05,099 INFO [StoreOpener-365408da311d11f25467a308428c748a-1] regionserver.HStore(310): Store=365408da311d11f25467a308428c748a/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:05,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a 2023-07-12 15:05:05,100 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a 2023-07-12 15:05:05,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:05:05,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 365408da311d11f25467a308428c748a 2023-07-12 15:05:05,104 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 365408da311d11f25467a308428c748a; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62673515, jitterRate=-0.06609185039997101}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:05:05,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 365408da311d11f25467a308428c748a: 2023-07-12 15:05:05,105 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a., pid=35, masterSystemTime=1689174305087 2023-07-12 15:05:05,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:05,107 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:05,107 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:05,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d50f055275b35092de8260c021ea4888, NAME => 'testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888.', STARTKEY => 'a', ENDKEY => 'b'} 2023-07-12 15:05:05,107 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=365408da311d11f25467a308428c748a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:05,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 d50f055275b35092de8260c021ea4888 2023-07-12 15:05:05,108 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:05,108 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689174305107"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174305107"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174305107"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174305107"}]},"ts":"1689174305107"} 2023-07-12 15:05:05,108 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for d50f055275b35092de8260c021ea4888 2023-07-12 15:05:05,108 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for d50f055275b35092de8260c021ea4888 2023-07-12 15:05:05,109 INFO [StoreOpener-d50f055275b35092de8260c021ea4888-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region d50f055275b35092de8260c021ea4888 2023-07-12 15:05:05,110 DEBUG [StoreOpener-d50f055275b35092de8260c021ea4888-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888/f1 2023-07-12 15:05:05,110 DEBUG [StoreOpener-d50f055275b35092de8260c021ea4888-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888/f1 2023-07-12 15:05:05,110 INFO [StoreOpener-d50f055275b35092de8260c021ea4888-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d50f055275b35092de8260c021ea4888 columnFamilyName f1 2023-07-12 15:05:05,111 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=35, resume processing ppid=28 2023-07-12 15:05:05,111 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, ppid=28, state=SUCCESS; OpenRegionProcedure 365408da311d11f25467a308428c748a, server=jenkins-hbase5.apache.org,35497,1689174298592 in 163 msec 2023-07-12 15:05:05,111 INFO [StoreOpener-d50f055275b35092de8260c021ea4888-1] regionserver.HStore(310): Store=d50f055275b35092de8260c021ea4888/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:05,111 INFO [StoreOpener-d50f055275b35092de8260c021ea4888-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region d50f055275b35092de8260c021ea4888 2023-07-12 15:05:05,112 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=365408da311d11f25467a308428c748a, ASSIGN in 334 msec 2023-07-12 15:05:05,112 DEBUG [StoreOpener-d50f055275b35092de8260c021ea4888-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888/f2 2023-07-12 15:05:05,112 DEBUG [StoreOpener-d50f055275b35092de8260c021ea4888-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888/f2 2023-07-12 15:05:05,112 INFO [StoreOpener-d50f055275b35092de8260c021ea4888-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d50f055275b35092de8260c021ea4888 columnFamilyName f2 2023-07-12 15:05:05,113 INFO [StoreOpener-d50f055275b35092de8260c021ea4888-1] regionserver.HStore(310): Store=d50f055275b35092de8260c021ea4888/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:05,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888 2023-07-12 15:05:05,115 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888 2023-07-12 15:05:05,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:05:05,118 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for d50f055275b35092de8260c021ea4888 2023-07-12 15:05:05,118 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened d50f055275b35092de8260c021ea4888; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59948036, jitterRate=-0.10670465230941772}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:05:05,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for d50f055275b35092de8260c021ea4888: 2023-07-12 15:05:05,119 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888., pid=34, masterSystemTime=1689174305087 2023-07-12 15:05:05,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:05,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:05,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:05,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => bad388b7482b4b27d67bc96cd80a265f, NAME => 'testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f.', STARTKEY => 'b', ENDKEY => 'c'} 2023-07-12 15:05:05,121 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=d50f055275b35092de8260c021ea4888, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:05,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:05,121 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174305121"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174305121"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174305121"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174305121"}]},"ts":"1689174305121"} 2023-07-12 15:05:05,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:05,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:05,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:05,124 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=29 2023-07-12 15:05:05,124 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=29, state=SUCCESS; OpenRegionProcedure d50f055275b35092de8260c021ea4888, server=jenkins-hbase5.apache.org,35497,1689174298592 in 181 msec 2023-07-12 15:05:05,125 INFO [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:05,125 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=d50f055275b35092de8260c021ea4888, ASSIGN in 347 msec 2023-07-12 15:05:05,126 DEBUG [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f/f1 2023-07-12 15:05:05,126 DEBUG [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f/f1 2023-07-12 15:05:05,126 INFO [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bad388b7482b4b27d67bc96cd80a265f columnFamilyName f1 2023-07-12 15:05:05,127 INFO [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] regionserver.HStore(310): Store=bad388b7482b4b27d67bc96cd80a265f/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:05,127 INFO [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:05,128 DEBUG [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f/f2 2023-07-12 15:05:05,128 DEBUG [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f/f2 2023-07-12 15:05:05,129 INFO [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bad388b7482b4b27d67bc96cd80a265f columnFamilyName f2 2023-07-12 15:05:05,129 INFO [StoreOpener-bad388b7482b4b27d67bc96cd80a265f-1] regionserver.HStore(310): Store=bad388b7482b4b27d67bc96cd80a265f/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:05,130 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:05,132 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:05,133 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:05:05,134 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:05,135 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened bad388b7482b4b27d67bc96cd80a265f; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71410465, jitterRate=0.06409884989261627}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:05:05,135 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for bad388b7482b4b27d67bc96cd80a265f: 2023-07-12 15:05:05,136 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f., pid=32, masterSystemTime=1689174305087 2023-07-12 15:05:05,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:05,137 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:05,137 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:05,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ffe3dd29195363fb366cf4b7acdf928f, NAME => 'testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f.', STARTKEY => 'c', ENDKEY => ''} 2023-07-12 15:05:05,138 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=bad388b7482b4b27d67bc96cd80a265f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:05,138 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689174305138"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174305138"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174305138"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174305138"}]},"ts":"1689174305138"} 2023-07-12 15:05:05,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:05,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:05:05,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:05,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:05,139 INFO [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:05,140 DEBUG [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f/f1 2023-07-12 15:05:05,140 DEBUG [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f/f1 2023-07-12 15:05:05,141 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=30 2023-07-12 15:05:05,141 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=30, state=SUCCESS; OpenRegionProcedure bad388b7482b4b27d67bc96cd80a265f, server=jenkins-hbase5.apache.org,35497,1689174298592 in 204 msec 2023-07-12 15:05:05,141 INFO [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ffe3dd29195363fb366cf4b7acdf928f columnFamilyName f1 2023-07-12 15:05:05,141 INFO [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] regionserver.HStore(310): Store=ffe3dd29195363fb366cf4b7acdf928f/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:05,142 INFO [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:05,142 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=bad388b7482b4b27d67bc96cd80a265f, ASSIGN in 364 msec 2023-07-12 15:05:05,142 DEBUG [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f/f2 2023-07-12 15:05:05,142 DEBUG [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f/f2 2023-07-12 15:05:05,143 INFO [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ffe3dd29195363fb366cf4b7acdf928f columnFamilyName f2 2023-07-12 15:05:05,143 INFO [StoreOpener-ffe3dd29195363fb366cf4b7acdf928f-1] regionserver.HStore(310): Store=ffe3dd29195363fb366cf4b7acdf928f/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:05:05,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:05,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:05,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:05:05,149 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:05,150 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened ffe3dd29195363fb366cf4b7acdf928f; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74194671, jitterRate=0.10558675229549408}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:05:05,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for ffe3dd29195363fb366cf4b7acdf928f: 2023-07-12 15:05:05,150 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f., pid=33, masterSystemTime=1689174305087 2023-07-12 15:05:05,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:05,152 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:05,153 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=ffe3dd29195363fb366cf4b7acdf928f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:05,153 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689174305152"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689174305152"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689174305152"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689174305152"}]},"ts":"1689174305152"} 2023-07-12 15:05:05,155 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=31 2023-07-12 15:05:05,156 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=31, state=SUCCESS; OpenRegionProcedure ffe3dd29195363fb366cf4b7acdf928f, server=jenkins-hbase5.apache.org,35497,1689174298592 in 215 msec 2023-07-12 15:05:05,157 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=27 2023-07-12 15:05:05,157 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ffe3dd29195363fb366cf4b7acdf928f, ASSIGN in 379 msec 2023-07-12 15:05:05,158 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689174305158"}]},"ts":"1689174305158"} 2023-07-12 15:05:05,159 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=ENABLED in hbase:meta 2023-07-12 15:05:05,163 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testEnableTableWithFailoverAtStep3' was successfully enabled. 2023-07-12 15:05:05,165 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 in 2.0250 sec 2023-07-12 15:05:05,186 INFO [Listener at localhost/44677] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 15:05:05,186 INFO [Listener at localhost/44677] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 15:05:05,186 DEBUG [Listener at localhost/44677] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x05948539 to 127.0.0.1:51235 2023-07-12 15:05:05,186 DEBUG [Listener at localhost/44677] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:05:05,186 DEBUG [Listener at localhost/44677] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 15:05:05,186 DEBUG [Listener at localhost/44677] util.JVMClusterUtil(257): Found active master hash=1846745148, stopped=false 2023-07-12 15:05:05,186 INFO [Listener at localhost/44677] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase5.apache.org,46407,1689174298568 2023-07-12 15:05:05,189 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): regionserver:35497-0x101c5ad18690002, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 15:05:05,189 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 15:05:05,189 INFO [Listener at localhost/44677] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:05:05,189 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:05:05,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35497-0x101c5ad18690002, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:05:05,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:05:05,190 DEBUG [Listener at localhost/44677] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a94c521 to 127.0.0.1:51235 2023-07-12 15:05:05,190 DEBUG [Listener at localhost/44677] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:05:05,190 INFO [Listener at localhost/44677] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,35497,1689174298592' ***** 2023-07-12 15:05:05,190 INFO [Listener at localhost/44677] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 15:05:05,190 INFO [RS:0;jenkins-hbase5:35497] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 15:05:05,191 INFO [RS:0;jenkins-hbase5:35497] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 15:05:05,191 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 15:05:05,191 INFO [RS:0;jenkins-hbase5:35497] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 15:05:05,191 INFO [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(3305): Received CLOSE for b4ef2da9a8e7a43cdb82478409694269 2023-07-12 15:05:05,191 INFO [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(3305): Received CLOSE for d50f055275b35092de8260c021ea4888 2023-07-12 15:05:05,192 INFO [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(3305): Received CLOSE for 365408da311d11f25467a308428c748a 2023-07-12 15:05:05,192 INFO [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(3305): Received CLOSE for ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:05,192 INFO [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(3305): Received CLOSE for bad388b7482b4b27d67bc96cd80a265f 2023-07-12 15:05:05,192 INFO [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:05,192 DEBUG [RS:0;jenkins-hbase5:35497] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7fda9f2e to 127.0.0.1:51235 2023-07-12 15:05:05,192 DEBUG [RS:0;jenkins-hbase5:35497] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:05:05,192 INFO [RS:0;jenkins-hbase5:35497] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 15:05:05,192 INFO [RS:0;jenkins-hbase5:35497] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 15:05:05,192 INFO [RS:0;jenkins-hbase5:35497] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 15:05:05,193 INFO [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 15:05:05,192 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing b4ef2da9a8e7a43cdb82478409694269, disabling compactions & flushes 2023-07-12 15:05:05,193 INFO [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(1474): Waiting on 6 regions to close 2023-07-12 15:05:05,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269. 2023-07-12 15:05:05,193 DEBUG [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(1478): Online Regions={b4ef2da9a8e7a43cdb82478409694269=hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269., d50f055275b35092de8260c021ea4888=testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888., 365408da311d11f25467a308428c748a=testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a., ffe3dd29195363fb366cf4b7acdf928f=testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f., 1588230740=hbase:meta,,1.1588230740, bad388b7482b4b27d67bc96cd80a265f=testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f.} 2023-07-12 15:05:05,193 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 15:05:05,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269. 2023-07-12 15:05:05,193 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 15:05:05,193 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 15:05:05,193 DEBUG [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(1504): Waiting on 1588230740, 365408da311d11f25467a308428c748a, b4ef2da9a8e7a43cdb82478409694269, bad388b7482b4b27d67bc96cd80a265f, d50f055275b35092de8260c021ea4888, ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:05,193 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 15:05:05,194 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 15:05:05,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269. after waiting 0 ms 2023-07-12 15:05:05,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269. 2023-07-12 15:05:05,194 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=15.07 KB heapSize=24.05 KB 2023-07-12 15:05:05,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing b4ef2da9a8e7a43cdb82478409694269 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-12 15:05:05,212 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=14.53 KB at sequenceid=41 (bloomFilter=false), to=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/.tmp/info/236f89f352224d979ce95696318706e7 2023-07-12 15:05:05,251 INFO [regionserver/jenkins-hbase5:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 15:05:05,394 DEBUG [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(1504): Waiting on 1588230740, 365408da311d11f25467a308428c748a, b4ef2da9a8e7a43cdb82478409694269, bad388b7482b4b27d67bc96cd80a265f, d50f055275b35092de8260c021ea4888, ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:05,594 DEBUG [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(1504): Waiting on 1588230740, 365408da311d11f25467a308428c748a, b4ef2da9a8e7a43cdb82478409694269, bad388b7482b4b27d67bc96cd80a265f, d50f055275b35092de8260c021ea4888, ffe3dd29195363fb366cf4b7acdf928f 2023-07-12 15:05:05,611 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/namespace/b4ef2da9a8e7a43cdb82478409694269/.tmp/info/3d5bdb74559f4c91913ce7ee510eb17d 2023-07-12 15:05:05,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/namespace/b4ef2da9a8e7a43cdb82478409694269/.tmp/info/3d5bdb74559f4c91913ce7ee510eb17d as hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/namespace/b4ef2da9a8e7a43cdb82478409694269/info/3d5bdb74559f4c91913ce7ee510eb17d 2023-07-12 15:05:05,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/namespace/b4ef2da9a8e7a43cdb82478409694269/info/3d5bdb74559f4c91913ce7ee510eb17d, entries=2, sequenceid=6, filesize=4.8 K 2023-07-12 15:05:05,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for b4ef2da9a8e7a43cdb82478409694269 in 430ms, sequenceid=6, compaction requested=false 2023-07-12 15:05:05,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 15:05:05,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/namespace/b4ef2da9a8e7a43cdb82478409694269/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-12 15:05:05,633 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=556 B at sequenceid=41 (bloomFilter=false), to=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/.tmp/table/801a06a01fe547b290bc45ee16009b75 2023-07-12 15:05:05,634 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269. 2023-07-12 15:05:05,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for b4ef2da9a8e7a43cdb82478409694269: 2023-07-12 15:05:05,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689174300773.b4ef2da9a8e7a43cdb82478409694269. 2023-07-12 15:05:05,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing d50f055275b35092de8260c021ea4888, disabling compactions & flushes 2023-07-12 15:05:05,634 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:05,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:05,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. after waiting 0 ms 2023-07-12 15:05:05,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:05,637 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 15:05:05,638 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testEnableTableWithFailoverAtStep3' 2023-07-12 15:05:05,638 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/d50f055275b35092de8260c021ea4888/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-07-12 15:05:05,639 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:05,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for d50f055275b35092de8260c021ea4888: 2023-07-12 15:05:05,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testEnableTableWithFailoverAtStep3,a,1689174301324.d50f055275b35092de8260c021ea4888. 2023-07-12 15:05:05,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 365408da311d11f25467a308428c748a, disabling compactions & flushes 2023-07-12 15:05:05,639 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:05,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:05,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. after waiting 0 ms 2023-07-12 15:05:05,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:05,641 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/.tmp/info/236f89f352224d979ce95696318706e7 as hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/info/236f89f352224d979ce95696318706e7 2023-07-12 15:05:05,642 INFO [regionserver/jenkins-hbase5:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-12 15:05:05,642 INFO [regionserver/jenkins-hbase5:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-12 15:05:05,642 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/365408da311d11f25467a308428c748a/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-07-12 15:05:05,643 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:05,643 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 365408da311d11f25467a308428c748a: 2023-07-12 15:05:05,643 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testEnableTableWithFailoverAtStep3,,1689174301324.365408da311d11f25467a308428c748a. 2023-07-12 15:05:05,644 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing ffe3dd29195363fb366cf4b7acdf928f, disabling compactions & flushes 2023-07-12 15:05:05,644 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:05,644 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:05,644 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. after waiting 0 ms 2023-07-12 15:05:05,644 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:05,646 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/info/236f89f352224d979ce95696318706e7, entries=70, sequenceid=41, filesize=14.7 K 2023-07-12 15:05:05,647 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/.tmp/table/801a06a01fe547b290bc45ee16009b75 as hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/table/801a06a01fe547b290bc45ee16009b75 2023-07-12 15:05:05,648 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/ffe3dd29195363fb366cf4b7acdf928f/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-07-12 15:05:05,648 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:05,648 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for ffe3dd29195363fb366cf4b7acdf928f: 2023-07-12 15:05:05,648 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testEnableTableWithFailoverAtStep3,c,1689174301324.ffe3dd29195363fb366cf4b7acdf928f. 2023-07-12 15:05:05,649 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing bad388b7482b4b27d67bc96cd80a265f, disabling compactions & flushes 2023-07-12 15:05:05,649 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:05,649 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:05,649 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. after waiting 0 ms 2023-07-12 15:05:05,649 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:05,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/default/testEnableTableWithFailoverAtStep3/bad388b7482b4b27d67bc96cd80a265f/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-07-12 15:05:05,654 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/table/801a06a01fe547b290bc45ee16009b75, entries=5, sequenceid=41, filesize=4.9 K 2023-07-12 15:05:05,654 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~15.07 KB/15431, heapSize ~23.77 KB/24336, currentSize=0 B/0 for 1588230740 in 460ms, sequenceid=41, compaction requested=false 2023-07-12 15:05:05,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:05,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for bad388b7482b4b27d67bc96cd80a265f: 2023-07-12 15:05:05,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testEnableTableWithFailoverAtStep3,b,1689174301324.bad388b7482b4b27d67bc96cd80a265f. 2023-07-12 15:05:05,667 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/data/hbase/meta/1588230740/recovered.edits/44.seqid, newMaxSeqId=44, maxSeqId=1 2023-07-12 15:05:05,668 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 15:05:05,669 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 15:05:05,669 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 15:05:05,669 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 15:05:05,794 INFO [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,35497,1689174298592; all regions closed. 2023-07-12 15:05:05,800 DEBUG [RS:0;jenkins-hbase5:35497] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/oldWALs 2023-07-12 15:05:05,800 INFO [RS:0;jenkins-hbase5:35497] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C35497%2C1689174298592.meta:.meta(num 1689174300713) 2023-07-12 15:05:05,804 DEBUG [RS:0;jenkins-hbase5:35497] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/oldWALs 2023-07-12 15:05:05,804 INFO [RS:0;jenkins-hbase5:35497] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C35497%2C1689174298592:(num 1689174299773) 2023-07-12 15:05:05,804 DEBUG [RS:0;jenkins-hbase5:35497] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:05:05,804 INFO [RS:0;jenkins-hbase5:35497] regionserver.LeaseManager(133): Closed leases 2023-07-12 15:05:05,804 INFO [RS:0;jenkins-hbase5:35497] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase5:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 15:05:05,805 INFO [regionserver/jenkins-hbase5:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:05:05,806 INFO [RS:0;jenkins-hbase5:35497] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:35497 2023-07-12 15:05:05,808 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): regionserver:35497-0x101c5ad18690002, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase5.apache.org,35497,1689174298592 2023-07-12 15:05:05,808 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): regionserver:35497-0x101c5ad18690002, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:05:05,808 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:05:05,810 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase5.apache.org,35497,1689174298592] 2023-07-12 15:05:05,810 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase5.apache.org,35497,1689174298592; numProcessing=1 2023-07-12 15:05:05,811 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase5.apache.org,35497,1689174298592 already deleted, retry=false 2023-07-12 15:05:05,812 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase5.apache.org,35497,1689174298592 expired; onlineServers=0 2023-07-12 15:05:05,812 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,46407,1689174298568' ***** 2023-07-12 15:05:05,812 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-12 15:05:05,812 DEBUG [M:1;jenkins-hbase5:46407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a2c9126, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:05:05,812 INFO [M:1;jenkins-hbase5:46407] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,46407,1689174298568 2023-07-12 15:05:05,812 INFO [M:1;jenkins-hbase5:46407] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,46407,1689174298568; all regions closed. 2023-07-12 15:05:05,812 DEBUG [M:1;jenkins-hbase5:46407] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:05:05,812 DEBUG [M:1;jenkins-hbase5:46407] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 15:05:05,812 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 15:05:05,812 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174304742] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689174304742,5,FailOnTimeoutGroup] 2023-07-12 15:05:05,812 DEBUG [M:1;jenkins-hbase5:46407] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 15:05:05,812 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174304741] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689174304741,5,FailOnTimeoutGroup] 2023-07-12 15:05:05,813 INFO [M:1;jenkins-hbase5:46407] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 15:05:05,813 INFO [M:1;jenkins-hbase5:46407] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 15:05:05,813 INFO [M:1;jenkins-hbase5:46407] hbase.ChoreService(369): Chore service for: master/jenkins-hbase5:0 had [] on shutdown 2023-07-12 15:05:05,813 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:05:05,813 DEBUG [M:1;jenkins-hbase5:46407] master.HMaster(1512): Stopping service threads 2023-07-12 15:05:05,813 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:05:05,813 INFO [M:1;jenkins-hbase5:46407] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 15:05:05,813 ERROR [M:1;jenkins-hbase5:46407] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 15:05:05,814 INFO [M:1;jenkins-hbase5:46407] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:05:05,814 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 15:05:05,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:05:05,814 DEBUG [M:1;jenkins-hbase5:46407] zookeeper.ZKUtil(398): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 15:05:05,814 WARN [M:1;jenkins-hbase5:46407] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 15:05:05,814 INFO [M:1;jenkins-hbase5:46407] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:05:05,815 INFO [M:1;jenkins-hbase5:46407] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-12 15:05:05,815 DEBUG [M:1;jenkins-hbase5:46407] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:05:05,815 INFO [M:1;jenkins-hbase5:46407] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:05:05,815 DEBUG [M:1;jenkins-hbase5:46407] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:05:05,815 DEBUG [M:1;jenkins-hbase5:46407] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:05:05,815 DEBUG [M:1;jenkins-hbase5:46407] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:05:05,815 INFO [M:1;jenkins-hbase5:46407] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=24.96 KB heapSize=30.73 KB 2023-07-12 15:05:05,910 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): regionserver:35497-0x101c5ad18690002, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:05:05,910 INFO [RS:0;jenkins-hbase5:35497] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,35497,1689174298592; zookeeper connection closed. 2023-07-12 15:05:05,910 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): regionserver:35497-0x101c5ad18690002, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:05:05,911 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@13e33d68] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@13e33d68 2023-07-12 15:05:05,911 INFO [Listener at localhost/44677] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 1 regionserver(s) complete 2023-07-12 15:05:06,224 INFO [M:1;jenkins-hbase5:46407] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.96 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/fe0c976a65ea418898d8208de7cc2915 2023-07-12 15:05:06,230 DEBUG [M:1;jenkins-hbase5:46407] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/fe0c976a65ea418898d8208de7cc2915 as hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/fe0c976a65ea418898d8208de7cc2915 2023-07-12 15:05:06,235 INFO [M:1;jenkins-hbase5:46407] regionserver.HStore(1080): Added hdfs://localhost:40551/user/jenkins/test-data/6ee39092-8203-5ee4-5f04-98714549e919/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/fe0c976a65ea418898d8208de7cc2915, entries=9, sequenceid=254, filesize=5.4 K 2023-07-12 15:05:06,236 INFO [M:1;jenkins-hbase5:46407] regionserver.HRegion(2948): Finished flush of dataSize ~24.96 KB/25554, heapSize ~30.71 KB/31448, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 421ms, sequenceid=254, compaction requested=false 2023-07-12 15:05:06,237 INFO [M:1;jenkins-hbase5:46407] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:05:06,237 DEBUG [M:1;jenkins-hbase5:46407] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:05:06,241 INFO [M:1;jenkins-hbase5:46407] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 15:05:06,241 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:05:06,242 INFO [M:1;jenkins-hbase5:46407] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:46407 2023-07-12 15:05:06,245 DEBUG [M:1;jenkins-hbase5:46407] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase5.apache.org,46407,1689174298568 already deleted, retry=false 2023-07-12 15:05:06,347 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:05:06,347 INFO [M:1;jenkins-hbase5:46407] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,46407,1689174298568; zookeeper connection closed. 2023-07-12 15:05:06,347 DEBUG [Listener at localhost/44677-EventThread] zookeeper.ZKWatcher(600): master:46407-0x101c5ad18690001, quorum=127.0.0.1:51235, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:05:06,348 WARN [Listener at localhost/44677] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 15:05:06,359 INFO [Listener at localhost/44677] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 15:05:06,464 WARN [BP-1485170651-172.31.10.131-1689174297619 heartbeating to localhost/127.0.0.1:40551] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 15:05:06,464 WARN [BP-1485170651-172.31.10.131-1689174297619 heartbeating to localhost/127.0.0.1:40551] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1485170651-172.31.10.131-1689174297619 (Datanode Uuid 592f0b78-afac-481f-be23-d1838c16fb33) service to localhost/127.0.0.1:40551 2023-07-12 15:05:06,465 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/cluster_de344728-3c1b-401b-1994-b1d21d451bc9/dfs/data/data1/current/BP-1485170651-172.31.10.131-1689174297619] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 15:05:06,465 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/baac7eae-1cee-8351-f02f-ec5b851bbed0/cluster_de344728-3c1b-401b-1994-b1d21d451bc9/dfs/data/data2/current/BP-1485170651-172.31.10.131-1689174297619] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 15:05:06,477 INFO [Listener at localhost/44677] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 15:05:06,590 INFO [Listener at localhost/44677] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 15:05:06,599 INFO [Listener at localhost/44677] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-07-12 15:05:06,611 INFO [Listener at localhost/44677] hbase.ResourceChecker(175): after: master.procedure.TestMasterFailoverWithProcedures#testEnableTableWithFailover Thread=150 (was 125) Potentially hanging thread: Listener at localhost/44677 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-17-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (593222724) connection to localhost/127.0.0.1:40551 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-15-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.4@localhost:40551 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-17-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (593222724) connection to localhost/127.0.0.1:40551 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-17-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (593222724) connection to localhost/127.0.0.1:40551 from jenkins.hfs.4 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: LeaseRenewer:jenkins@localhost:40551 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=515 (was 487) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=200 (was 184) - SystemLoadAverage LEAK? -, ProcessCount=161 (was 161), AvailableMemoryMB=7938 (was 7722) - AvailableMemoryMB LEAK? -