2023-07-11 14:20:09,771 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures timeout: 13 mins 2023-07-11 14:20:10,002 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832 2023-07-11 14:20:10,033 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.procedure.TestMasterFailoverWithProcedures#testDisableTableWithFailover Thread=10, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=385, ProcessCount=171, AvailableMemoryMB=8318 2023-07-11 14:20:10,040 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=2, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-11 14:20:10,040 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/cluster_acd48a5f-1519-3a78-5e80-8455ee69e37a, deleteOnExit=true 2023-07-11 14:20:10,041 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-11 14:20:10,041 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/test.cache.data in system properties and HBase conf 2023-07-11 14:20:10,042 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/hadoop.tmp.dir in system properties and HBase conf 2023-07-11 14:20:10,042 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/hadoop.log.dir in system properties and HBase conf 2023-07-11 14:20:10,043 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-11 14:20:10,043 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-11 14:20:10,043 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-11 14:20:10,158 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-11 14:20:10,607 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-11 14:20:10,611 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-11 14:20:10,611 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-11 14:20:10,612 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-11 14:20:10,612 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-11 14:20:10,613 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-11 14:20:10,613 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-11 14:20:10,614 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-11 14:20:10,614 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-11 14:20:10,615 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-11 14:20:10,615 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/nfs.dump.dir in system properties and HBase conf 2023-07-11 14:20:10,616 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/java.io.tmpdir in system properties and HBase conf 2023-07-11 14:20:10,616 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-11 14:20:10,617 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-11 14:20:10,617 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-11 14:20:11,149 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-11 14:20:11,154 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-11 14:20:11,436 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-11 14:20:11,608 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-11 14:20:11,623 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-11 14:20:11,660 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-11 14:20:11,719 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/java.io.tmpdir/Jetty_localhost_44757_hdfs____.p9uu2l/webapp 2023-07-11 14:20:11,842 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44757 2023-07-11 14:20:11,852 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-11 14:20:11,852 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-11 14:20:12,312 WARN [Listener at localhost/46725] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-11 14:20:12,421 WARN [Listener at localhost/46725] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-11 14:20:12,462 WARN [Listener at localhost/46725] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-11 14:20:12,470 INFO [Listener at localhost/46725] log.Slf4jLog(67): jetty-6.1.26 2023-07-11 14:20:12,477 INFO [Listener at localhost/46725] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/java.io.tmpdir/Jetty_localhost_44721_datanode____.9qt5nn/webapp 2023-07-11 14:20:12,594 INFO [Listener at localhost/46725] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44721 2023-07-11 14:20:12,975 WARN [Listener at localhost/43533] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-11 14:20:13,445 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x32c80986106dcfe3: Processing first storage report for DS-960c0af4-f4f5-4bac-8f7c-1e8a57a16cc2 from datanode b127a362-8312-4dd6-bceb-e37ed2152684 2023-07-11 14:20:13,447 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x32c80986106dcfe3: from storage DS-960c0af4-f4f5-4bac-8f7c-1e8a57a16cc2 node DatanodeRegistration(127.0.0.1:45455, datanodeUuid=b127a362-8312-4dd6-bceb-e37ed2152684, infoPort=39677, infoSecurePort=0, ipcPort=43533, storageInfo=lv=-57;cid=testClusterID;nsid=1603760294;c=1689085211229), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-11 14:20:13,447 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x32c80986106dcfe3: Processing first storage report for DS-a7ceb7e9-5cb7-47d4-965e-03105452cd6f from datanode b127a362-8312-4dd6-bceb-e37ed2152684 2023-07-11 14:20:13,447 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x32c80986106dcfe3: from storage DS-a7ceb7e9-5cb7-47d4-965e-03105452cd6f node DatanodeRegistration(127.0.0.1:45455, datanodeUuid=b127a362-8312-4dd6-bceb-e37ed2152684, infoPort=39677, infoSecurePort=0, ipcPort=43533, storageInfo=lv=-57;cid=testClusterID;nsid=1603760294;c=1689085211229), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-11 14:20:13,455 DEBUG [Listener at localhost/43533] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832 2023-07-11 14:20:13,517 INFO [Listener at localhost/43533] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/cluster_acd48a5f-1519-3a78-5e80-8455ee69e37a/zookeeper_0, clientPort=50917, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/cluster_acd48a5f-1519-3a78-5e80-8455ee69e37a/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/cluster_acd48a5f-1519-3a78-5e80-8455ee69e37a/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-11 14:20:13,530 INFO [Listener at localhost/43533] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=50917 2023-07-11 14:20:13,542 INFO [Listener at localhost/43533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:13,545 INFO [Listener at localhost/43533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:14,200 INFO [Listener at localhost/43533] util.FSUtils(471): Created version file at hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975 with version=8 2023-07-11 14:20:14,200 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/hbase-staging 2023-07-11 14:20:14,523 INFO [Listener at localhost/43533] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-11 14:20:14,969 INFO [Listener at localhost/43533] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-07-11 14:20:15,000 INFO [Listener at localhost/43533] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:15,001 INFO [Listener at localhost/43533] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:15,001 INFO [Listener at localhost/43533] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 14:20:15,001 INFO [Listener at localhost/43533] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:15,001 INFO [Listener at localhost/43533] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 14:20:15,141 INFO [Listener at localhost/43533] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 14:20:15,211 DEBUG [Listener at localhost/43533] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-11 14:20:15,303 INFO [Listener at localhost/43533] ipc.NettyRpcServer(120): Bind to /172.31.2.10:36993 2023-07-11 14:20:15,312 INFO [Listener at localhost/43533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:15,314 INFO [Listener at localhost/43533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:15,335 INFO [Listener at localhost/43533] zookeeper.RecoverableZooKeeper(93): Process identifier=master:36993 connecting to ZooKeeper ensemble=127.0.0.1:50917 2023-07-11 14:20:15,380 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:369930x0, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 14:20:15,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:36993-0x10154b3bede0000 connected 2023-07-11 14:20:15,408 DEBUG [Listener at localhost/43533] zookeeper.ZKUtil(164): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:15,409 DEBUG [Listener at localhost/43533] zookeeper.ZKUtil(164): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:15,412 DEBUG [Listener at localhost/43533] zookeeper.ZKUtil(164): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 14:20:15,420 DEBUG [Listener at localhost/43533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36993 2023-07-11 14:20:15,421 DEBUG [Listener at localhost/43533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36993 2023-07-11 14:20:15,421 DEBUG [Listener at localhost/43533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36993 2023-07-11 14:20:15,421 DEBUG [Listener at localhost/43533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36993 2023-07-11 14:20:15,422 DEBUG [Listener at localhost/43533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36993 2023-07-11 14:20:15,427 INFO [Listener at localhost/43533] master.HMaster(444): hbase.rootdir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975, hbase.cluster.distributed=false 2023-07-11 14:20:15,464 INFO [Listener at localhost/43533] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-07-11 14:20:15,464 INFO [Listener at localhost/43533] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:15,464 INFO [Listener at localhost/43533] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:15,464 INFO [Listener at localhost/43533] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 14:20:15,464 INFO [Listener at localhost/43533] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:15,464 INFO [Listener at localhost/43533] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 14:20:15,465 INFO [Listener at localhost/43533] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 14:20:15,466 INFO [Listener at localhost/43533] ipc.NettyRpcServer(120): Bind to /172.31.2.10:34133 2023-07-11 14:20:15,466 INFO [Listener at localhost/43533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:15,468 INFO [Listener at localhost/43533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:15,470 INFO [Listener at localhost/43533] zookeeper.RecoverableZooKeeper(93): Process identifier=master:34133 connecting to ZooKeeper ensemble=127.0.0.1:50917 2023-07-11 14:20:15,475 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:341330x0, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 14:20:15,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:34133-0x10154b3bede0001 connected 2023-07-11 14:20:15,482 DEBUG [Listener at localhost/43533] zookeeper.ZKUtil(164): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:15,483 DEBUG [Listener at localhost/43533] zookeeper.ZKUtil(164): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:15,483 DEBUG [Listener at localhost/43533] zookeeper.ZKUtil(164): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 14:20:15,484 DEBUG [Listener at localhost/43533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34133 2023-07-11 14:20:15,484 DEBUG [Listener at localhost/43533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34133 2023-07-11 14:20:15,484 DEBUG [Listener at localhost/43533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34133 2023-07-11 14:20:15,485 DEBUG [Listener at localhost/43533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34133 2023-07-11 14:20:15,485 DEBUG [Listener at localhost/43533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34133 2023-07-11 14:20:15,485 INFO [Listener at localhost/43533] master.HMaster(444): hbase.rootdir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975, hbase.cluster.distributed=false 2023-07-11 14:20:15,521 INFO [Listener at localhost/43533] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-07-11 14:20:15,521 INFO [Listener at localhost/43533] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:15,522 INFO [Listener at localhost/43533] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:15,522 INFO [Listener at localhost/43533] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 14:20:15,522 INFO [Listener at localhost/43533] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:15,522 INFO [Listener at localhost/43533] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 14:20:15,527 INFO [Listener at localhost/43533] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 14:20:15,531 INFO [Listener at localhost/43533] ipc.NettyRpcServer(120): Bind to /172.31.2.10:35917 2023-07-11 14:20:15,534 INFO [Listener at localhost/43533] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-11 14:20:15,542 DEBUG [Listener at localhost/43533] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-11 14:20:15,543 INFO [Listener at localhost/43533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:15,546 INFO [Listener at localhost/43533] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:15,548 INFO [Listener at localhost/43533] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35917 connecting to ZooKeeper ensemble=127.0.0.1:50917 2023-07-11 14:20:15,552 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): regionserver:359170x0, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 14:20:15,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:35917-0x10154b3bede0002 connected 2023-07-11 14:20:15,553 DEBUG [Listener at localhost/43533] zookeeper.ZKUtil(164): regionserver:35917-0x10154b3bede0002, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:15,554 DEBUG [Listener at localhost/43533] zookeeper.ZKUtil(164): regionserver:35917-0x10154b3bede0002, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:15,554 DEBUG [Listener at localhost/43533] zookeeper.ZKUtil(164): regionserver:35917-0x10154b3bede0002, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 14:20:15,555 DEBUG [Listener at localhost/43533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35917 2023-07-11 14:20:15,555 DEBUG [Listener at localhost/43533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35917 2023-07-11 14:20:15,555 DEBUG [Listener at localhost/43533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35917 2023-07-11 14:20:15,556 DEBUG [Listener at localhost/43533] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35917 2023-07-11 14:20:15,556 DEBUG [Listener at localhost/43533] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35917 2023-07-11 14:20:15,558 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,34133,1689085215463 2023-07-11 14:20:15,558 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,36993,1689085214354 2023-07-11 14:20:15,568 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:15,568 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:15,568 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,34133,1689085215463 2023-07-11 14:20:15,569 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,36993,1689085214354 2023-07-11 14:20:15,592 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:15,592 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): regionserver:35917-0x10154b3bede0002, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:15,592 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:15,592 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:15,592 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:15,593 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:15,593 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:15,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:15,594 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,36993,1689085214354 from backup master directory 2023-07-11 14:20:15,594 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase9.apache.org,36993,1689085214354; waiting to become the next active master 2023-07-11 14:20:15,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:15,598 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:15,598 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,36993,1689085214354 2023-07-11 14:20:15,598 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:15,599 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 14:20:15,599 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase9.apache.org,36993,1689085214354 2023-07-11 14:20:15,601 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-11 14:20:15,603 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-11 14:20:15,682 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/hbase.id with ID: 01861c03-e532-4f8c-98fb-fd761389e3ea 2023-07-11 14:20:15,723 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:15,737 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:15,737 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:15,775 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x03e9f0d4 to 127.0.0.1:50917 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:20:15,811 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@161bc3b0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:20:15,836 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-11 14:20:15,838 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-11 14:20:15,855 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-11 14:20:15,855 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-11 14:20:15,857 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-11 14:20:15,860 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-11 14:20:15,861 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:20:15,894 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store-tmp 2023-07-11 14:20:15,919 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:15,920 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 14:20:15,920 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:15,920 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:15,920 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 14:20:15,920 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:15,920 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:15,920 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:15,922 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/WALs/jenkins-hbase9.apache.org,36993,1689085214354 2023-07-11 14:20:15,942 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C36993%2C1689085214354, suffix=, logDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/WALs/jenkins-hbase9.apache.org,36993,1689085214354, archiveDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/oldWALs, maxLogs=10 2023-07-11 14:20:15,993 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45455,DS-960c0af4-f4f5-4bac-8f7c-1e8a57a16cc2,DISK] 2023-07-11 14:20:16,000 DEBUG [RS-EventLoopGroup-4-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-11 14:20:16,060 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/WALs/jenkins-hbase9.apache.org,36993,1689085214354/jenkins-hbase9.apache.org%2C36993%2C1689085214354.1689085215952 2023-07-11 14:20:16,060 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45455,DS-960c0af4-f4f5-4bac-8f7c-1e8a57a16cc2,DISK]] 2023-07-11 14:20:16,061 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:16,061 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:16,064 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:16,065 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:16,119 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:16,126 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-11 14:20:16,149 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-11 14:20:16,160 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:16,165 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:16,167 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:16,182 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:16,186 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:16,187 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72819012, jitterRate=0.0850878357887268}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 14:20:16,187 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:16,189 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-11 14:20:16,207 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:16,207 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:16,210 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:16,211 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-11 14:20:16,244 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-07-11 14:20:16,244 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-11 14:20:16,269 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-11 14:20:16,274 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-11 14:20:16,297 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-11 14:20:16,300 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-11 14:20:16,301 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-11 14:20:16,306 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-11 14:20:16,310 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-11 14:20:16,314 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:16,314 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:16,316 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-11 14:20:16,316 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-11 14:20:16,327 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-11 14:20:16,331 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:16,331 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): regionserver:35917-0x10154b3bede0002, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:16,331 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:16,331 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:16,332 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:16,332 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase9.apache.org,36993,1689085214354, sessionid=0x10154b3bede0000, setting cluster-up flag (Was=false) 2023-07-11 14:20:16,334 INFO [M:1;jenkins-hbase9:34133] regionserver.HRegionServer(951): ClusterId : 01861c03-e532-4f8c-98fb-fd761389e3ea 2023-07-11 14:20:16,346 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:16,346 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:16,351 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-11 14:20:16,352 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,36993,1689085214354 2023-07-11 14:20:16,357 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:16,357 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:16,362 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-11 14:20:16,363 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,36993,1689085214354 2023-07-11 14:20:16,366 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.hbase-snapshot/.tmp 2023-07-11 14:20:16,461 INFO [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer(951): ClusterId : 01861c03-e532-4f8c-98fb-fd761389e3ea 2023-07-11 14:20:16,466 DEBUG [RS:0;jenkins-hbase9:35917] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-11 14:20:16,471 DEBUG [RS:0;jenkins-hbase9:35917] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-11 14:20:16,471 DEBUG [RS:0;jenkins-hbase9:35917] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-11 14:20:16,474 DEBUG [RS:0;jenkins-hbase9:35917] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-11 14:20:16,476 DEBUG [RS:0;jenkins-hbase9:35917] zookeeper.ReadOnlyZKClient(139): Connect 0x2dda002c to 127.0.0.1:50917 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:20:16,481 DEBUG [RS:0;jenkins-hbase9:35917] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@231debe7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:20:16,482 DEBUG [RS:0;jenkins-hbase9:35917] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27226b5c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-11 14:20:16,497 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:16,505 DEBUG [RS:0;jenkins-hbase9:35917] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase9:35917 2023-07-11 14:20:16,509 INFO [RS:0;jenkins-hbase9:35917] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-11 14:20:16,509 INFO [RS:0;jenkins-hbase9:35917] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-11 14:20:16,509 DEBUG [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer(1022): About to register with Master. 2023-07-11 14:20:16,509 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:16,510 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:16,510 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:16,510 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:16,510 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-07-11 14:20:16,510 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:16,511 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-11 14:20:16,511 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:16,512 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085246512 2023-07-11 14:20:16,512 INFO [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase9.apache.org,36993,1689085214354 with isa=jenkins-hbase9.apache.org/172.31.2.10:35917, startcode=1689085215520 2023-07-11 14:20:16,514 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-11 14:20:16,520 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:16,520 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-11 14:20:16,525 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-11 14:20:16,527 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-11 14:20:16,533 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-11 14:20:16,533 DEBUG [RS:0;jenkins-hbase9:35917] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-11 14:20:16,534 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-11 14:20:16,534 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-11 14:20:16,534 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-11 14:20:16,535 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:16,538 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-11 14:20:16,539 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-11 14:20:16,539 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-11 14:20:16,541 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-11 14:20:16,542 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-11 14:20:16,544 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085216543,5,FailOnTimeoutGroup] 2023-07-11 14:20:16,544 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085216544,5,FailOnTimeoutGroup] 2023-07-11 14:20:16,544 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:16,544 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-11 14:20:16,546 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:16,546 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:16,557 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:16,558 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:16,558 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975 2023-07-11 14:20:16,575 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:16,578 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-11 14:20:16,582 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/info 2023-07-11 14:20:16,583 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-11 14:20:16,584 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:16,584 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-11 14:20:16,587 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/rep_barrier 2023-07-11 14:20:16,588 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-11 14:20:16,589 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:16,590 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-11 14:20:16,592 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/table 2023-07-11 14:20:16,592 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-11 14:20:16,593 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:16,595 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740 2023-07-11 14:20:16,596 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740 2023-07-11 14:20:16,599 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:45161, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-11 14:20:16,600 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-11 14:20:16,602 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-11 14:20:16,605 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:16,606 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63832000, jitterRate=-0.048829078674316406}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-11 14:20:16,606 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-11 14:20:16,608 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-11 14:20:16,608 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-11 14:20:16,608 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-11 14:20:16,608 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-11 14:20:16,608 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-11 14:20:16,610 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-11 14:20:16,610 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-11 14:20:16,614 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36993] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:16,616 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:16,616 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-11 14:20:16,627 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-11 14:20:16,634 DEBUG [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975 2023-07-11 14:20:16,634 DEBUG [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:46725 2023-07-11 14:20:16,634 DEBUG [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-11 14:20:16,639 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 14:20:16,640 DEBUG [RS:0;jenkins-hbase9:35917] zookeeper.ZKUtil(162): regionserver:35917-0x10154b3bede0002, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:16,640 WARN [RS:0;jenkins-hbase9:35917] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 14:20:16,640 INFO [RS:0;jenkins-hbase9:35917] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:20:16,641 DEBUG [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer(1948): logDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/WALs/jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:16,641 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,35917,1689085215520] 2023-07-11 14:20:16,642 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-11 14:20:16,645 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-11 14:20:16,650 DEBUG [RS:0;jenkins-hbase9:35917] zookeeper.ZKUtil(162): regionserver:35917-0x10154b3bede0002, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:16,660 DEBUG [RS:0;jenkins-hbase9:35917] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-11 14:20:16,669 INFO [RS:0;jenkins-hbase9:35917] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-11 14:20:16,691 INFO [RS:0;jenkins-hbase9:35917] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-11 14:20:16,694 INFO [RS:0;jenkins-hbase9:35917] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-11 14:20:16,695 INFO [RS:0;jenkins-hbase9:35917] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:16,696 INFO [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-11 14:20:16,703 INFO [RS:0;jenkins-hbase9:35917] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:16,703 DEBUG [RS:0;jenkins-hbase9:35917] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:16,703 DEBUG [RS:0;jenkins-hbase9:35917] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:16,703 DEBUG [RS:0;jenkins-hbase9:35917] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:16,703 DEBUG [RS:0;jenkins-hbase9:35917] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:16,703 DEBUG [RS:0;jenkins-hbase9:35917] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:16,703 DEBUG [RS:0;jenkins-hbase9:35917] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-11 14:20:16,703 DEBUG [RS:0;jenkins-hbase9:35917] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:16,703 DEBUG [RS:0;jenkins-hbase9:35917] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:16,704 DEBUG [RS:0;jenkins-hbase9:35917] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:16,704 DEBUG [RS:0;jenkins-hbase9:35917] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:16,705 INFO [RS:0;jenkins-hbase9:35917] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:16,705 INFO [RS:0;jenkins-hbase9:35917] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:16,705 INFO [RS:0;jenkins-hbase9:35917] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:16,721 INFO [RS:0;jenkins-hbase9:35917] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-11 14:20:16,723 INFO [RS:0;jenkins-hbase9:35917] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,35917,1689085215520-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:16,739 INFO [RS:0;jenkins-hbase9:35917] regionserver.Replication(203): jenkins-hbase9.apache.org,35917,1689085215520 started 2023-07-11 14:20:16,739 INFO [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer(1637): Serving as jenkins-hbase9.apache.org,35917,1689085215520, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:35917, sessionid=0x10154b3bede0002 2023-07-11 14:20:16,739 DEBUG [RS:0;jenkins-hbase9:35917] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-11 14:20:16,739 DEBUG [RS:0;jenkins-hbase9:35917] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:16,740 DEBUG [RS:0;jenkins-hbase9:35917] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,35917,1689085215520' 2023-07-11 14:20:16,740 DEBUG [RS:0;jenkins-hbase9:35917] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-11 14:20:16,740 DEBUG [RS:0;jenkins-hbase9:35917] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-11 14:20:16,741 DEBUG [RS:0;jenkins-hbase9:35917] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-11 14:20:16,741 DEBUG [RS:0;jenkins-hbase9:35917] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-11 14:20:16,741 DEBUG [RS:0;jenkins-hbase9:35917] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:16,741 DEBUG [RS:0;jenkins-hbase9:35917] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,35917,1689085215520' 2023-07-11 14:20:16,741 DEBUG [RS:0;jenkins-hbase9:35917] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-11 14:20:16,741 DEBUG [RS:0;jenkins-hbase9:35917] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-11 14:20:16,742 DEBUG [RS:0;jenkins-hbase9:35917] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-11 14:20:16,742 INFO [RS:0;jenkins-hbase9:35917] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-11 14:20:16,742 INFO [RS:0;jenkins-hbase9:35917] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-11 14:20:16,797 DEBUG [jenkins-hbase9:36993] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-11 14:20:16,799 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,35917,1689085215520, state=OPENING 2023-07-11 14:20:16,807 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-11 14:20:16,809 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:16,809 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:16,809 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:20:16,809 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:20:16,813 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,35917,1689085215520}] 2023-07-11 14:20:16,852 INFO [RS:0;jenkins-hbase9:35917] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C35917%2C1689085215520, suffix=, logDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/WALs/jenkins-hbase9.apache.org,35917,1689085215520, archiveDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/oldWALs, maxLogs=32 2023-07-11 14:20:16,872 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45455,DS-960c0af4-f4f5-4bac-8f7c-1e8a57a16cc2,DISK] 2023-07-11 14:20:16,878 INFO [RS:0;jenkins-hbase9:35917] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/WALs/jenkins-hbase9.apache.org,35917,1689085215520/jenkins-hbase9.apache.org%2C35917%2C1689085215520.1689085216855 2023-07-11 14:20:16,878 DEBUG [RS:0;jenkins-hbase9:35917] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45455,DS-960c0af4-f4f5-4bac-8f7c-1e8a57a16cc2,DISK]] 2023-07-11 14:20:16,995 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:16,998 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-11 14:20:17,001 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:36594, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-11 14:20:17,013 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-11 14:20:17,013 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:20:17,017 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C35917%2C1689085215520.meta, suffix=.meta, logDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/WALs/jenkins-hbase9.apache.org,35917,1689085215520, archiveDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/oldWALs, maxLogs=32 2023-07-11 14:20:17,034 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45455,DS-960c0af4-f4f5-4bac-8f7c-1e8a57a16cc2,DISK] 2023-07-11 14:20:17,038 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/WALs/jenkins-hbase9.apache.org,35917,1689085215520/jenkins-hbase9.apache.org%2C35917%2C1689085215520.meta.1689085217018.meta 2023-07-11 14:20:17,038 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45455,DS-960c0af4-f4f5-4bac-8f7c-1e8a57a16cc2,DISK]] 2023-07-11 14:20:17,038 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:17,040 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-11 14:20:17,056 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-11 14:20:17,061 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-11 14:20:17,066 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-11 14:20:17,066 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:17,066 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-11 14:20:17,067 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-11 14:20:17,069 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-11 14:20:17,071 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/info 2023-07-11 14:20:17,071 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/info 2023-07-11 14:20:17,072 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-11 14:20:17,073 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:17,073 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-11 14:20:17,074 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/rep_barrier 2023-07-11 14:20:17,074 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/rep_barrier 2023-07-11 14:20:17,075 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-11 14:20:17,076 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:17,076 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-11 14:20:17,077 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/table 2023-07-11 14:20:17,077 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/table 2023-07-11 14:20:17,078 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-11 14:20:17,079 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:17,081 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740 2023-07-11 14:20:17,084 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740 2023-07-11 14:20:17,088 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-11 14:20:17,091 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-11 14:20:17,093 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66225467, jitterRate=-0.013163641095161438}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-11 14:20:17,093 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-11 14:20:17,104 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689085216988 2023-07-11 14:20:17,121 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-11 14:20:17,121 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-11 14:20:17,122 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,35917,1689085215520, state=OPEN 2023-07-11 14:20:17,127 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-11 14:20:17,127 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-11 14:20:17,127 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:20:17,127 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:20:17,131 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-11 14:20:17,131 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,35917,1689085215520 in 314 msec 2023-07-11 14:20:17,136 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-11 14:20:17,136 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 506 msec 2023-07-11 14:20:17,142 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 719 msec 2023-07-11 14:20:17,142 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689085217142, completionTime=-1 2023-07-11 14:20:17,142 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-11 14:20:17,143 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:17,205 DEBUG [hconnection-0x5d52e59e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 14:20:17,209 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:36602, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 14:20:17,227 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:17,227 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085277227 2023-07-11 14:20:17,227 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085337227 2023-07-11 14:20:17,228 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 84 msec 2023-07-11 14:20:17,249 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,36993,1689085214354-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:17,250 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,36993,1689085214354-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:17,250 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,36993,1689085214354-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:17,252 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:36993, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:17,253 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:17,259 DEBUG [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-11 14:20:17,267 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-11 14:20:17,268 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-11 14:20:17,277 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:17,280 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-11 14:20:17,282 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-11 14:20:17,303 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/hbase/namespace/a48689024a2a1f1a7a4accb24d7943d7 2023-07-11 14:20:17,305 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/hbase/namespace/a48689024a2a1f1a7a4accb24d7943d7 empty. 2023-07-11 14:20:17,306 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/hbase/namespace/a48689024a2a1f1a7a4accb24d7943d7 2023-07-11 14:20:17,306 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-11 14:20:17,328 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:17,330 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => a48689024a2a1f1a7a4accb24d7943d7, NAME => 'hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp 2023-07-11 14:20:17,341 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:17,342 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing a48689024a2a1f1a7a4accb24d7943d7, disabling compactions & flushes 2023-07-11 14:20:17,342 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7. 2023-07-11 14:20:17,342 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7. 2023-07-11 14:20:17,342 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7. after waiting 0 ms 2023-07-11 14:20:17,342 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7. 2023-07-11 14:20:17,342 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7. 2023-07-11 14:20:17,342 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for a48689024a2a1f1a7a4accb24d7943d7: 2023-07-11 14:20:17,346 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-11 14:20:17,361 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689085217349"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085217349"}]},"ts":"1689085217349"} 2023-07-11 14:20:17,388 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-11 14:20:17,390 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-11 14:20:17,395 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085217391"}]},"ts":"1689085217391"} 2023-07-11 14:20:17,399 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-11 14:20:17,405 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a48689024a2a1f1a7a4accb24d7943d7, ASSIGN}] 2023-07-11 14:20:17,409 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a48689024a2a1f1a7a4accb24d7943d7, ASSIGN 2023-07-11 14:20:17,410 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=a48689024a2a1f1a7a4accb24d7943d7, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,35917,1689085215520; forceNewPlan=false, retain=false 2023-07-11 14:20:17,561 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=a48689024a2a1f1a7a4accb24d7943d7, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:17,562 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689085217561"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085217561"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085217561"}]},"ts":"1689085217561"} 2023-07-11 14:20:17,566 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure a48689024a2a1f1a7a4accb24d7943d7, server=jenkins-hbase9.apache.org,35917,1689085215520}] 2023-07-11 14:20:17,726 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7. 2023-07-11 14:20:17,726 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a48689024a2a1f1a7a4accb24d7943d7, NAME => 'hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:17,727 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace a48689024a2a1f1a7a4accb24d7943d7 2023-07-11 14:20:17,728 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:17,728 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for a48689024a2a1f1a7a4accb24d7943d7 2023-07-11 14:20:17,728 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for a48689024a2a1f1a7a4accb24d7943d7 2023-07-11 14:20:17,729 INFO [StoreOpener-a48689024a2a1f1a7a4accb24d7943d7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region a48689024a2a1f1a7a4accb24d7943d7 2023-07-11 14:20:17,732 DEBUG [StoreOpener-a48689024a2a1f1a7a4accb24d7943d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/namespace/a48689024a2a1f1a7a4accb24d7943d7/info 2023-07-11 14:20:17,732 DEBUG [StoreOpener-a48689024a2a1f1a7a4accb24d7943d7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/namespace/a48689024a2a1f1a7a4accb24d7943d7/info 2023-07-11 14:20:17,732 INFO [StoreOpener-a48689024a2a1f1a7a4accb24d7943d7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a48689024a2a1f1a7a4accb24d7943d7 columnFamilyName info 2023-07-11 14:20:17,733 INFO [StoreOpener-a48689024a2a1f1a7a4accb24d7943d7-1] regionserver.HStore(310): Store=a48689024a2a1f1a7a4accb24d7943d7/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:17,735 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/namespace/a48689024a2a1f1a7a4accb24d7943d7 2023-07-11 14:20:17,736 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/namespace/a48689024a2a1f1a7a4accb24d7943d7 2023-07-11 14:20:17,742 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for a48689024a2a1f1a7a4accb24d7943d7 2023-07-11 14:20:17,746 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/namespace/a48689024a2a1f1a7a4accb24d7943d7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:17,747 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened a48689024a2a1f1a7a4accb24d7943d7; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75475619, jitterRate=0.12467436492443085}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 14:20:17,747 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for a48689024a2a1f1a7a4accb24d7943d7: 2023-07-11 14:20:17,749 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7., pid=6, masterSystemTime=1689085217720 2023-07-11 14:20:17,752 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7. 2023-07-11 14:20:17,752 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7. 2023-07-11 14:20:17,754 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=a48689024a2a1f1a7a4accb24d7943d7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:17,754 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689085217753"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085217753"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085217753"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085217753"}]},"ts":"1689085217753"} 2023-07-11 14:20:17,767 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-11 14:20:17,767 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure a48689024a2a1f1a7a4accb24d7943d7, server=jenkins-hbase9.apache.org,35917,1689085215520 in 194 msec 2023-07-11 14:20:17,770 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-11 14:20:17,771 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=a48689024a2a1f1a7a4accb24d7943d7, ASSIGN in 362 msec 2023-07-11 14:20:17,772 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-11 14:20:17,772 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085217772"}]},"ts":"1689085217772"} 2023-07-11 14:20:17,776 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-11 14:20:17,781 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-11 14:20:17,781 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-11 14:20:17,783 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-11 14:20:17,783 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:17,783 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:17,784 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 512 msec 2023-07-11 14:20:17,818 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:17,836 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-11 14:20:17,843 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 33 msec 2023-07-11 14:20:17,852 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:17,864 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-11 14:20:17,868 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-07-11 14:20:17,878 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-11 14:20:17,881 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-11 14:20:17,881 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.282sec 2023-07-11 14:20:17,883 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-11 14:20:17,885 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-11 14:20:17,885 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-11 14:20:17,886 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,36993,1689085214354-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-11 14:20:17,886 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,36993,1689085214354-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-11 14:20:17,897 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-11 14:20:17,966 DEBUG [Listener at localhost/43533] zookeeper.ReadOnlyZKClient(139): Connect 0x07e87950 to 127.0.0.1:50917 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:20:17,971 DEBUG [Listener at localhost/43533] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69109578, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:20:17,983 DEBUG [hconnection-0x2539d01e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 14:20:17,995 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:36618, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 14:20:18,005 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase9.apache.org,36993,1689085214354 2023-07-11 14:20:18,007 WARN [Listener at localhost/43533] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-11 14:20:18,011 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:18,013 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-11 14:20:18,015 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-11 14:20:18,024 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/default/testDisableTableWithFailoverAtStep3/7678ac0cab253467f45346026a631b20 2023-07-11 14:20:18,024 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/default/testDisableTableWithFailoverAtStep3/74c58b571981d83dcf418f1080d591b9 2023-07-11 14:20:18,024 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/default/testDisableTableWithFailoverAtStep3/cdb24378120768c854e74a1b1338ae63 2023-07-11 14:20:18,024 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/default/testDisableTableWithFailoverAtStep3/9901b6faf1d3748db88cfd3a05b4468a 2023-07-11 14:20:18,025 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/default/testDisableTableWithFailoverAtStep3/7678ac0cab253467f45346026a631b20 empty. 2023-07-11 14:20:18,025 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/default/testDisableTableWithFailoverAtStep3/9901b6faf1d3748db88cfd3a05b4468a empty. 2023-07-11 14:20:18,026 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/default/testDisableTableWithFailoverAtStep3/74c58b571981d83dcf418f1080d591b9 empty. 2023-07-11 14:20:18,025 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/default/testDisableTableWithFailoverAtStep3/cdb24378120768c854e74a1b1338ae63 empty. 2023-07-11 14:20:18,027 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/default/testDisableTableWithFailoverAtStep3/7678ac0cab253467f45346026a631b20 2023-07-11 14:20:18,027 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/default/testDisableTableWithFailoverAtStep3/74c58b571981d83dcf418f1080d591b9 2023-07-11 14:20:18,027 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/default/testDisableTableWithFailoverAtStep3/9901b6faf1d3748db88cfd3a05b4468a 2023-07-11 14:20:18,027 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/default/testDisableTableWithFailoverAtStep3/cdb24378120768c854e74a1b1338ae63 2023-07-11 14:20:18,027 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testDisableTableWithFailoverAtStep3 regions 2023-07-11 14:20:18,447 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp/data/default/testDisableTableWithFailoverAtStep3/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:18,449 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7678ac0cab253467f45346026a631b20, NAME => 'testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20.', STARTKEY => '', ENDKEY => 'a'}, tableDescriptor='testDisableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp 2023-07-11 14:20:18,450 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(7675): creating {ENCODED => 9901b6faf1d3748db88cfd3a05b4468a, NAME => 'testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a.', STARTKEY => 'a', ENDKEY => 'b'}, tableDescriptor='testDisableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp 2023-07-11 14:20:18,450 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(7675): creating {ENCODED => 74c58b571981d83dcf418f1080d591b9, NAME => 'testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9.', STARTKEY => 'b', ENDKEY => 'c'}, tableDescriptor='testDisableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp 2023-07-11 14:20:18,490 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:18,494 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1604): Closing 9901b6faf1d3748db88cfd3a05b4468a, disabling compactions & flushes 2023-07-11 14:20:18,494 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a. 2023-07-11 14:20:18,494 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a. 2023-07-11 14:20:18,494 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a. after waiting 0 ms 2023-07-11 14:20:18,494 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a. 2023-07-11 14:20:18,494 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a. 2023-07-11 14:20:18,495 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1558): Region close journal for 9901b6faf1d3748db88cfd3a05b4468a: 2023-07-11 14:20:18,491 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:18,496 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1604): Closing 7678ac0cab253467f45346026a631b20, disabling compactions & flushes 2023-07-11 14:20:18,496 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:18,496 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20. 2023-07-11 14:20:18,496 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1604): Closing 74c58b571981d83dcf418f1080d591b9, disabling compactions & flushes 2023-07-11 14:20:18,496 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20. 2023-07-11 14:20:18,496 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9. 2023-07-11 14:20:18,497 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20. after waiting 0 ms 2023-07-11 14:20:18,497 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(7675): creating {ENCODED => cdb24378120768c854e74a1b1338ae63, NAME => 'testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63.', STARTKEY => 'c', ENDKEY => ''}, tableDescriptor='testDisableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.tmp 2023-07-11 14:20:18,497 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20. 2023-07-11 14:20:18,497 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9. 2023-07-11 14:20:18,497 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20. 2023-07-11 14:20:18,498 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9. after waiting 0 ms 2023-07-11 14:20:18,498 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1558): Region close journal for 7678ac0cab253467f45346026a631b20: 2023-07-11 14:20:18,498 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9. 2023-07-11 14:20:18,499 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9. 2023-07-11 14:20:18,499 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1558): Region close journal for 74c58b571981d83dcf418f1080d591b9: 2023-07-11 14:20:18,513 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:18,514 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1604): Closing cdb24378120768c854e74a1b1338ae63, disabling compactions & flushes 2023-07-11 14:20:18,514 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63. 2023-07-11 14:20:18,514 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63. 2023-07-11 14:20:18,514 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63. after waiting 0 ms 2023-07-11 14:20:18,514 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63. 2023-07-11 14:20:18,514 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63. 2023-07-11 14:20:18,514 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1558): Region close journal for cdb24378120768c854e74a1b1338ae63: 2023-07-11 14:20:18,519 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_ADD_TO_META 2023-07-11 14:20:18,521 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689085218521"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085218521"}]},"ts":"1689085218521"} 2023-07-11 14:20:18,522 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085218521"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085218521"}]},"ts":"1689085218521"} 2023-07-11 14:20:18,522 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689085218521"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085218521"}]},"ts":"1689085218521"} 2023-07-11 14:20:18,522 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085218521"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085218521"}]},"ts":"1689085218521"} 2023-07-11 14:20:18,570 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 4 regions to meta. 2023-07-11 14:20:18,571 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-11 14:20:18,572 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDisableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085218572"}]},"ts":"1689085218572"} 2023-07-11 14:20:18,574 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDisableTableWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-11 14:20:18,578 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7678ac0cab253467f45346026a631b20, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=9901b6faf1d3748db88cfd3a05b4468a, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=74c58b571981d83dcf418f1080d591b9, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=cdb24378120768c854e74a1b1338ae63, ASSIGN}] 2023-07-11 14:20:18,582 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=cdb24378120768c854e74a1b1338ae63, ASSIGN 2023-07-11 14:20:18,583 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=cdb24378120768c854e74a1b1338ae63, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,35917,1689085215520; forceNewPlan=false, retain=false 2023-07-11 14:20:18,584 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=74c58b571981d83dcf418f1080d591b9, ASSIGN 2023-07-11 14:20:18,585 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=74c58b571981d83dcf418f1080d591b9, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,35917,1689085215520; forceNewPlan=false, retain=false 2023-07-11 14:20:18,586 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=9901b6faf1d3748db88cfd3a05b4468a, ASSIGN 2023-07-11 14:20:18,587 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=9901b6faf1d3748db88cfd3a05b4468a, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,35917,1689085215520; forceNewPlan=false, retain=false 2023-07-11 14:20:18,588 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7678ac0cab253467f45346026a631b20, ASSIGN 2023-07-11 14:20:18,589 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7678ac0cab253467f45346026a631b20, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,35917,1689085215520; forceNewPlan=false, retain=false 2023-07-11 14:20:18,734 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=74c58b571981d83dcf418f1080d591b9, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:18,734 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689085218734"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085218734"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085218734"}]},"ts":"1689085218734"} 2023-07-11 14:20:18,737 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=12, state=RUNNABLE; OpenRegionProcedure 74c58b571981d83dcf418f1080d591b9, server=jenkins-hbase9.apache.org,35917,1689085215520}] 2023-07-11 14:20:18,742 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=9901b6faf1d3748db88cfd3a05b4468a, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:18,742 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689085218742"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085218742"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085218742"}]},"ts":"1689085218742"} 2023-07-11 14:20:18,745 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=11, state=RUNNABLE; OpenRegionProcedure 9901b6faf1d3748db88cfd3a05b4468a, server=jenkins-hbase9.apache.org,35917,1689085215520}] 2023-07-11 14:20:18,749 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=7678ac0cab253467f45346026a631b20, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:18,749 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085218749"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085218749"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085218749"}]},"ts":"1689085218749"} 2023-07-11 14:20:18,752 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=10, state=RUNNABLE; OpenRegionProcedure 7678ac0cab253467f45346026a631b20, server=jenkins-hbase9.apache.org,35917,1689085215520}] 2023-07-11 14:20:18,756 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=cdb24378120768c854e74a1b1338ae63, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:18,756 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085218755"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085218755"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085218755"}]},"ts":"1689085218755"} 2023-07-11 14:20:18,758 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=13, state=RUNNABLE; OpenRegionProcedure cdb24378120768c854e74a1b1338ae63, server=jenkins-hbase9.apache.org,35917,1689085215520}] 2023-07-11 14:20:18,897 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20. 2023-07-11 14:20:18,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7678ac0cab253467f45346026a631b20, NAME => 'testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20.', STARTKEY => '', ENDKEY => 'a'} 2023-07-11 14:20:18,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDisableTableWithFailoverAtStep3 7678ac0cab253467f45346026a631b20 2023-07-11 14:20:18,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:18,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 7678ac0cab253467f45346026a631b20 2023-07-11 14:20:18,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 7678ac0cab253467f45346026a631b20 2023-07-11 14:20:18,900 INFO [StoreOpener-7678ac0cab253467f45346026a631b20-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 7678ac0cab253467f45346026a631b20 2023-07-11 14:20:18,902 DEBUG [StoreOpener-7678ac0cab253467f45346026a631b20-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/7678ac0cab253467f45346026a631b20/f1 2023-07-11 14:20:18,902 DEBUG [StoreOpener-7678ac0cab253467f45346026a631b20-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/7678ac0cab253467f45346026a631b20/f1 2023-07-11 14:20:18,903 INFO [StoreOpener-7678ac0cab253467f45346026a631b20-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7678ac0cab253467f45346026a631b20 columnFamilyName f1 2023-07-11 14:20:18,904 INFO [StoreOpener-7678ac0cab253467f45346026a631b20-1] regionserver.HStore(310): Store=7678ac0cab253467f45346026a631b20/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:18,904 INFO [StoreOpener-7678ac0cab253467f45346026a631b20-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 7678ac0cab253467f45346026a631b20 2023-07-11 14:20:18,906 DEBUG [StoreOpener-7678ac0cab253467f45346026a631b20-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/7678ac0cab253467f45346026a631b20/f2 2023-07-11 14:20:18,906 DEBUG [StoreOpener-7678ac0cab253467f45346026a631b20-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/7678ac0cab253467f45346026a631b20/f2 2023-07-11 14:20:18,906 INFO [StoreOpener-7678ac0cab253467f45346026a631b20-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7678ac0cab253467f45346026a631b20 columnFamilyName f2 2023-07-11 14:20:18,907 INFO [StoreOpener-7678ac0cab253467f45346026a631b20-1] regionserver.HStore(310): Store=7678ac0cab253467f45346026a631b20/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:18,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/7678ac0cab253467f45346026a631b20 2023-07-11 14:20:18,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/7678ac0cab253467f45346026a631b20 2023-07-11 14:20:18,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testDisableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:20:18,914 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 7678ac0cab253467f45346026a631b20 2023-07-11 14:20:18,919 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/7678ac0cab253467f45346026a631b20/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:18,919 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 7678ac0cab253467f45346026a631b20; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64241205, jitterRate=-0.042731449007987976}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:20:18,919 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 7678ac0cab253467f45346026a631b20: 2023-07-11 14:20:18,921 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20., pid=16, masterSystemTime=1689085218890 2023-07-11 14:20:18,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20. 2023-07-11 14:20:18,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20. 2023-07-11 14:20:18,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9. 2023-07-11 14:20:18,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 74c58b571981d83dcf418f1080d591b9, NAME => 'testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9.', STARTKEY => 'b', ENDKEY => 'c'} 2023-07-11 14:20:18,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDisableTableWithFailoverAtStep3 74c58b571981d83dcf418f1080d591b9 2023-07-11 14:20:18,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:18,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 74c58b571981d83dcf418f1080d591b9 2023-07-11 14:20:18,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 74c58b571981d83dcf418f1080d591b9 2023-07-11 14:20:18,924 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=7678ac0cab253467f45346026a631b20, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:18,925 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085218924"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085218924"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085218924"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085218924"}]},"ts":"1689085218924"} 2023-07-11 14:20:18,926 INFO [StoreOpener-74c58b571981d83dcf418f1080d591b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 74c58b571981d83dcf418f1080d591b9 2023-07-11 14:20:18,928 DEBUG [StoreOpener-74c58b571981d83dcf418f1080d591b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/74c58b571981d83dcf418f1080d591b9/f1 2023-07-11 14:20:18,928 DEBUG [StoreOpener-74c58b571981d83dcf418f1080d591b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/74c58b571981d83dcf418f1080d591b9/f1 2023-07-11 14:20:18,929 INFO [StoreOpener-74c58b571981d83dcf418f1080d591b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 74c58b571981d83dcf418f1080d591b9 columnFamilyName f1 2023-07-11 14:20:18,930 INFO [StoreOpener-74c58b571981d83dcf418f1080d591b9-1] regionserver.HStore(310): Store=74c58b571981d83dcf418f1080d591b9/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:18,930 INFO [StoreOpener-74c58b571981d83dcf418f1080d591b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 74c58b571981d83dcf418f1080d591b9 2023-07-11 14:20:18,932 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=10 2023-07-11 14:20:18,933 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=10, state=SUCCESS; OpenRegionProcedure 7678ac0cab253467f45346026a631b20, server=jenkins-hbase9.apache.org,35917,1689085215520 in 176 msec 2023-07-11 14:20:18,933 DEBUG [StoreOpener-74c58b571981d83dcf418f1080d591b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/74c58b571981d83dcf418f1080d591b9/f2 2023-07-11 14:20:18,934 DEBUG [StoreOpener-74c58b571981d83dcf418f1080d591b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/74c58b571981d83dcf418f1080d591b9/f2 2023-07-11 14:20:18,934 INFO [StoreOpener-74c58b571981d83dcf418f1080d591b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 74c58b571981d83dcf418f1080d591b9 columnFamilyName f2 2023-07-11 14:20:18,935 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7678ac0cab253467f45346026a631b20, ASSIGN in 355 msec 2023-07-11 14:20:18,935 INFO [StoreOpener-74c58b571981d83dcf418f1080d591b9-1] regionserver.HStore(310): Store=74c58b571981d83dcf418f1080d591b9/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:18,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/74c58b571981d83dcf418f1080d591b9 2023-07-11 14:20:18,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/74c58b571981d83dcf418f1080d591b9 2023-07-11 14:20:18,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testDisableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:20:18,943 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 74c58b571981d83dcf418f1080d591b9 2023-07-11 14:20:18,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/74c58b571981d83dcf418f1080d591b9/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:18,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 74c58b571981d83dcf418f1080d591b9; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68372121, jitterRate=0.01882399618625641}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:20:18,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 74c58b571981d83dcf418f1080d591b9: 2023-07-11 14:20:18,951 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9., pid=14, masterSystemTime=1689085218890 2023-07-11 14:20:18,954 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9. 2023-07-11 14:20:18,954 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9. 2023-07-11 14:20:18,954 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a. 2023-07-11 14:20:18,954 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9901b6faf1d3748db88cfd3a05b4468a, NAME => 'testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a.', STARTKEY => 'a', ENDKEY => 'b'} 2023-07-11 14:20:18,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDisableTableWithFailoverAtStep3 9901b6faf1d3748db88cfd3a05b4468a 2023-07-11 14:20:18,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:18,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 9901b6faf1d3748db88cfd3a05b4468a 2023-07-11 14:20:18,955 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=74c58b571981d83dcf418f1080d591b9, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:18,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 9901b6faf1d3748db88cfd3a05b4468a 2023-07-11 14:20:18,956 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689085218955"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085218955"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085218955"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085218955"}]},"ts":"1689085218955"} 2023-07-11 14:20:18,957 INFO [StoreOpener-9901b6faf1d3748db88cfd3a05b4468a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 9901b6faf1d3748db88cfd3a05b4468a 2023-07-11 14:20:18,960 DEBUG [StoreOpener-9901b6faf1d3748db88cfd3a05b4468a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/9901b6faf1d3748db88cfd3a05b4468a/f1 2023-07-11 14:20:18,960 DEBUG [StoreOpener-9901b6faf1d3748db88cfd3a05b4468a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/9901b6faf1d3748db88cfd3a05b4468a/f1 2023-07-11 14:20:18,960 INFO [StoreOpener-9901b6faf1d3748db88cfd3a05b4468a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9901b6faf1d3748db88cfd3a05b4468a columnFamilyName f1 2023-07-11 14:20:18,961 INFO [StoreOpener-9901b6faf1d3748db88cfd3a05b4468a-1] regionserver.HStore(310): Store=9901b6faf1d3748db88cfd3a05b4468a/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:18,962 INFO [StoreOpener-9901b6faf1d3748db88cfd3a05b4468a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 9901b6faf1d3748db88cfd3a05b4468a 2023-07-11 14:20:18,962 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=12 2023-07-11 14:20:18,963 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=12, state=SUCCESS; OpenRegionProcedure 74c58b571981d83dcf418f1080d591b9, server=jenkins-hbase9.apache.org,35917,1689085215520 in 222 msec 2023-07-11 14:20:18,965 DEBUG [StoreOpener-9901b6faf1d3748db88cfd3a05b4468a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/9901b6faf1d3748db88cfd3a05b4468a/f2 2023-07-11 14:20:18,965 DEBUG [StoreOpener-9901b6faf1d3748db88cfd3a05b4468a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/9901b6faf1d3748db88cfd3a05b4468a/f2 2023-07-11 14:20:18,966 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=74c58b571981d83dcf418f1080d591b9, ASSIGN in 385 msec 2023-07-11 14:20:18,966 INFO [StoreOpener-9901b6faf1d3748db88cfd3a05b4468a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9901b6faf1d3748db88cfd3a05b4468a columnFamilyName f2 2023-07-11 14:20:18,967 INFO [StoreOpener-9901b6faf1d3748db88cfd3a05b4468a-1] regionserver.HStore(310): Store=9901b6faf1d3748db88cfd3a05b4468a/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:18,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/9901b6faf1d3748db88cfd3a05b4468a 2023-07-11 14:20:18,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/9901b6faf1d3748db88cfd3a05b4468a 2023-07-11 14:20:18,972 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testDisableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:20:18,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 9901b6faf1d3748db88cfd3a05b4468a 2023-07-11 14:20:18,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/9901b6faf1d3748db88cfd3a05b4468a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:18,977 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 9901b6faf1d3748db88cfd3a05b4468a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59981261, jitterRate=-0.10620956122875214}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:20:18,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 9901b6faf1d3748db88cfd3a05b4468a: 2023-07-11 14:20:18,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a., pid=15, masterSystemTime=1689085218890 2023-07-11 14:20:18,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a. 2023-07-11 14:20:18,981 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a. 2023-07-11 14:20:18,981 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63. 2023-07-11 14:20:18,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => cdb24378120768c854e74a1b1338ae63, NAME => 'testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63.', STARTKEY => 'c', ENDKEY => ''} 2023-07-11 14:20:18,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDisableTableWithFailoverAtStep3 cdb24378120768c854e74a1b1338ae63 2023-07-11 14:20:18,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:18,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for cdb24378120768c854e74a1b1338ae63 2023-07-11 14:20:18,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for cdb24378120768c854e74a1b1338ae63 2023-07-11 14:20:18,982 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=9901b6faf1d3748db88cfd3a05b4468a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:18,982 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689085218981"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085218981"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085218981"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085218981"}]},"ts":"1689085218981"} 2023-07-11 14:20:18,983 INFO [StoreOpener-cdb24378120768c854e74a1b1338ae63-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region cdb24378120768c854e74a1b1338ae63 2023-07-11 14:20:18,985 DEBUG [StoreOpener-cdb24378120768c854e74a1b1338ae63-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/cdb24378120768c854e74a1b1338ae63/f1 2023-07-11 14:20:18,985 DEBUG [StoreOpener-cdb24378120768c854e74a1b1338ae63-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/cdb24378120768c854e74a1b1338ae63/f1 2023-07-11 14:20:18,986 INFO [StoreOpener-cdb24378120768c854e74a1b1338ae63-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cdb24378120768c854e74a1b1338ae63 columnFamilyName f1 2023-07-11 14:20:18,987 INFO [StoreOpener-cdb24378120768c854e74a1b1338ae63-1] regionserver.HStore(310): Store=cdb24378120768c854e74a1b1338ae63/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:18,987 INFO [StoreOpener-cdb24378120768c854e74a1b1338ae63-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region cdb24378120768c854e74a1b1338ae63 2023-07-11 14:20:18,988 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=11 2023-07-11 14:20:18,988 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=11, state=SUCCESS; OpenRegionProcedure 9901b6faf1d3748db88cfd3a05b4468a, server=jenkins-hbase9.apache.org,35917,1689085215520 in 240 msec 2023-07-11 14:20:18,990 DEBUG [StoreOpener-cdb24378120768c854e74a1b1338ae63-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/cdb24378120768c854e74a1b1338ae63/f2 2023-07-11 14:20:18,990 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=9901b6faf1d3748db88cfd3a05b4468a, ASSIGN in 410 msec 2023-07-11 14:20:18,990 DEBUG [StoreOpener-cdb24378120768c854e74a1b1338ae63-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/cdb24378120768c854e74a1b1338ae63/f2 2023-07-11 14:20:18,992 INFO [StoreOpener-cdb24378120768c854e74a1b1338ae63-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cdb24378120768c854e74a1b1338ae63 columnFamilyName f2 2023-07-11 14:20:18,993 INFO [StoreOpener-cdb24378120768c854e74a1b1338ae63-1] regionserver.HStore(310): Store=cdb24378120768c854e74a1b1338ae63/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:18,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/cdb24378120768c854e74a1b1338ae63 2023-07-11 14:20:18,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/cdb24378120768c854e74a1b1338ae63 2023-07-11 14:20:18,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testDisableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:20:19,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for cdb24378120768c854e74a1b1338ae63 2023-07-11 14:20:19,003 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/cdb24378120768c854e74a1b1338ae63/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:19,004 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened cdb24378120768c854e74a1b1338ae63; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70228236, jitterRate=0.04648226499557495}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:20:19,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for cdb24378120768c854e74a1b1338ae63: 2023-07-11 14:20:19,005 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63., pid=17, masterSystemTime=1689085218890 2023-07-11 14:20:19,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63. 2023-07-11 14:20:19,007 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63. 2023-07-11 14:20:19,008 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=cdb24378120768c854e74a1b1338ae63, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:19,008 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085219007"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085219007"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085219007"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085219007"}]},"ts":"1689085219007"} 2023-07-11 14:20:19,013 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=13 2023-07-11 14:20:19,013 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=13, state=SUCCESS; OpenRegionProcedure cdb24378120768c854e74a1b1338ae63, server=jenkins-hbase9.apache.org,35917,1689085215520 in 252 msec 2023-07-11 14:20:19,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=9 2023-07-11 14:20:19,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=cdb24378120768c854e74a1b1338ae63, ASSIGN in 435 msec 2023-07-11 14:20:19,017 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-11 14:20:19,017 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDisableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085219017"}]},"ts":"1689085219017"} 2023-07-11 14:20:19,019 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDisableTableWithFailoverAtStep3, state=ENABLED in hbase:meta 2023-07-11 14:20:19,022 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_POST_OPERATION 2023-07-11 14:20:19,025 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 in 1.0130 sec 2023-07-11 14:20:19,262 WARN [Listener at localhost/43533] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-07-11 14:20:19,269 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:19,272 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:19,272 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:19,272 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:19,272 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:19,519 INFO [Listener at localhost/43533] procedure.TestMasterFailoverWithProcedures(306): Restart 0 exec state: pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:19,520 INFO [Listener at localhost/43533] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:20:19,520 INFO [Listener at localhost/43533] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:19,521 INFO [Listener at localhost/43533] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:19,522 ERROR [Listener at localhost/43533] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] 2023-07-11 14:20:19,523 INFO [Listener at localhost/43533] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:20:19,523 INFO [Listener at localhost/43533] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:19,523 INFO [Listener at localhost/43533] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:19,523 INFO [Listener at localhost/43533] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:19,524 INFO [Listener at localhost/43533] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-11 14:20:19,527 DEBUG [Listener at localhost/43533] store.ProcedureTree(114): Procedure Procedure(pid=18, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DisableTableProcedure) stack ids=[] 2023-07-11 14:20:19,528 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:19,529 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:19,529 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:19,529 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:19,530 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:19,530 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(468): Loading pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:19,530 DEBUG [Listener at localhost/43533] procedure2.Procedure(966): pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:19,531 INFO [Listener at localhost/43533] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-07-11 14:20:19,532 DEBUG [Listener at localhost/43533] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:19,532 DEBUG [Listener at localhost/43533] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,35917,1689085215520, table=hbase:meta, region=1588230740 2023-07-11 14:20:19,534 INFO [Listener at localhost/43533] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085249534 2023-07-11 14:20:19,535 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:19,535 DEBUG [Listener at localhost/43533] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:19,539 DEBUG [Listener at localhost/43533] assignment.RegionStateStore(147): Load hbase:meta entry region=a48689024a2a1f1a7a4accb24d7943d7, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:19,540 DEBUG [Listener at localhost/43533] assignment.RegionStateStore(147): Load hbase:meta entry region=7678ac0cab253467f45346026a631b20, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:19,540 DEBUG [Listener at localhost/43533] assignment.RegionStateStore(147): Load hbase:meta entry region=9901b6faf1d3748db88cfd3a05b4468a, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:19,541 DEBUG [Listener at localhost/43533] assignment.RegionStateStore(147): Load hbase:meta entry region=74c58b571981d83dcf418f1080d591b9, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:19,541 DEBUG [Listener at localhost/43533] assignment.RegionStateStore(147): Load hbase:meta entry region=cdb24378120768c854e74a1b1338ae63, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:19,541 INFO [Listener at localhost/43533] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:19,542 INFO [Listener at localhost/43533] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085279541 2023-07-11 14:20:19,542 INFO [Listener at localhost/43533] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085339542 2023-07-11 14:20:19,542 INFO [Listener at localhost/43533] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-07-11 14:20:19,542 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:20:19,544 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:19,544 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_SET_DISABLING_TABLE_STATE, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:19,544 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:19,544 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_SET_DISABLING_TABLE_STATE, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:19,792 INFO [Listener at localhost/43533] procedure.TestMasterFailoverWithProcedures(306): Restart 1 exec state: pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:19,792 INFO [Listener at localhost/43533] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:20:19,792 INFO [Listener at localhost/43533] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:19,792 INFO [Listener at localhost/43533] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:19,793 ERROR [Listener at localhost/43533] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:19,793 INFO [Listener at localhost/43533] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:20:19,793 INFO [Listener at localhost/43533] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:19,793 INFO [Listener at localhost/43533] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:19,793 INFO [Listener at localhost/43533] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:19,794 INFO [Listener at localhost/43533] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-11 14:20:19,796 DEBUG [Listener at localhost/43533] store.ProcedureTree(114): Procedure Procedure(pid=18, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DisableTableProcedure) stack ids=[0] 2023-07-11 14:20:19,796 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:19,797 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:19,797 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:19,798 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:19,798 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:19,798 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(468): Loading pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:19,799 DEBUG [Listener at localhost/43533] procedure2.Procedure(966): pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:19,799 INFO [Listener at localhost/43533] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-07-11 14:20:19,799 DEBUG [Listener at localhost/43533] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:19,800 DEBUG [Listener at localhost/43533] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,35917,1689085215520, table=hbase:meta, region=1588230740 2023-07-11 14:20:19,800 INFO [Listener at localhost/43533] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085249800 2023-07-11 14:20:19,801 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:19,801 DEBUG [Listener at localhost/43533] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:19,805 DEBUG [Listener at localhost/43533] assignment.RegionStateStore(147): Load hbase:meta entry region=a48689024a2a1f1a7a4accb24d7943d7, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:19,805 DEBUG [Listener at localhost/43533] assignment.RegionStateStore(147): Load hbase:meta entry region=7678ac0cab253467f45346026a631b20, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:19,806 DEBUG [Listener at localhost/43533] assignment.RegionStateStore(147): Load hbase:meta entry region=9901b6faf1d3748db88cfd3a05b4468a, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:19,806 DEBUG [Listener at localhost/43533] assignment.RegionStateStore(147): Load hbase:meta entry region=74c58b571981d83dcf418f1080d591b9, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:19,806 DEBUG [Listener at localhost/43533] assignment.RegionStateStore(147): Load hbase:meta entry region=cdb24378120768c854e74a1b1338ae63, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:19,806 INFO [Listener at localhost/43533] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:19,806 INFO [Listener at localhost/43533] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085279806 2023-07-11 14:20:19,807 INFO [Listener at localhost/43533] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085339807 2023-07-11 14:20:19,807 INFO [Listener at localhost/43533] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-07-11 14:20:19,807 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:20:19,809 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDisableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085219809"}]},"ts":"1689085219809"} 2023-07-11 14:20:19,811 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDisableTableWithFailoverAtStep3, state=DISABLING in hbase:meta 2023-07-11 14:20:19,813 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testDisableTableWithFailoverAtStep3 to state=DISABLING 2023-07-11 14:20:19,813 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:19,813 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_MARK_REGIONS_OFFLINE, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:19,813 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:19,813 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_MARK_REGIONS_OFFLINE, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:20,057 INFO [Listener at localhost/43533] procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state: pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:20,057 INFO [Listener at localhost/43533] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:20:20,057 INFO [Listener at localhost/43533] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:20,057 INFO [Listener at localhost/43533] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:20,057 ERROR [Listener at localhost/43533] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:20,058 INFO [Listener at localhost/43533] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:20:20,058 INFO [Listener at localhost/43533] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:20,058 INFO [Listener at localhost/43533] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:20,058 INFO [Listener at localhost/43533] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:20,059 INFO [Listener at localhost/43533] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-11 14:20:20,060 DEBUG [Listener at localhost/43533] store.ProcedureTree(114): Procedure Procedure(pid=18, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DisableTableProcedure) stack ids=[0, 1] 2023-07-11 14:20:20,061 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:20,061 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:20,061 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:20,062 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:20,062 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:20,062 DEBUG [Listener at localhost/43533] procedure2.ProcedureExecutor(468): Loading pid=18, state=RUNNABLE:DISABLE_TABLE_SET_DISABLING_TABLE_STATE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:20,062 DEBUG [Listener at localhost/43533] procedure2.Procedure(966): pid=18, state=RUNNABLE:DISABLE_TABLE_SET_DISABLING_TABLE_STATE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:20,063 INFO [Listener at localhost/43533] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 3 msec 2023-07-11 14:20:20,063 DEBUG [Listener at localhost/43533] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:20,064 DEBUG [Listener at localhost/43533] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,35917,1689085215520, table=hbase:meta, region=1588230740 2023-07-11 14:20:20,064 INFO [Listener at localhost/43533] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085250064 2023-07-11 14:20:20,064 DEBUG [Listener at localhost/43533] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:20,065 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:DISABLE_TABLE_SET_DISABLING_TABLE_STATE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:20,068 DEBUG [Listener at localhost/43533] assignment.RegionStateStore(147): Load hbase:meta entry region=a48689024a2a1f1a7a4accb24d7943d7, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:20,069 DEBUG [Listener at localhost/43533] assignment.RegionStateStore(147): Load hbase:meta entry region=7678ac0cab253467f45346026a631b20, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:20,069 DEBUG [Listener at localhost/43533] assignment.RegionStateStore(147): Load hbase:meta entry region=9901b6faf1d3748db88cfd3a05b4468a, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:20,069 DEBUG [Listener at localhost/43533] assignment.RegionStateStore(147): Load hbase:meta entry region=74c58b571981d83dcf418f1080d591b9, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:20,070 DEBUG [Listener at localhost/43533] assignment.RegionStateStore(147): Load hbase:meta entry region=cdb24378120768c854e74a1b1338ae63, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:20,070 INFO [Listener at localhost/43533] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:20,070 INFO [Listener at localhost/43533] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085280070 2023-07-11 14:20:20,070 INFO [Listener at localhost/43533] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085340070 2023-07-11 14:20:20,070 INFO [Listener at localhost/43533] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-07-11 14:20:20,070 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDisableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085220070"}]},"ts":"1689085220070"} 2023-07-11 14:20:20,073 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDisableTableWithFailoverAtStep3, state=DISABLING in hbase:meta 2023-07-11 14:20:20,075 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testDisableTableWithFailoverAtStep3 to state=DISABLING 2023-07-11 14:20:20,075 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:20:20,078 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7678ac0cab253467f45346026a631b20, UNASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=9901b6faf1d3748db88cfd3a05b4468a, UNASSIGN}, {pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=74c58b571981d83dcf418f1080d591b9, UNASSIGN}, {pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=cdb24378120768c854e74a1b1338ae63, UNASSIGN}] 2023-07-11 14:20:20,078 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:20,078 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=18, state=WAITING:DISABLE_TABLE_ADD_REPLICATION_BARRIER, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:20,078 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:20,078 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=WAITING:DISABLE_TABLE_ADD_REPLICATION_BARRIER, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:20,320 INFO [Listener at localhost/43533] procedure.TestMasterFailoverWithProcedures(313): Trigger master failover 2023-07-11 14:20:20,321 INFO [Listener at localhost/43533] hbase.MiniHBaseCluster(607): Aborting jenkins-hbase9.apache.org,36993,1689085214354 2023-07-11 14:20:20,325 ERROR [Listener at localhost/43533] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-11 14:20:20,325 ERROR [Listener at localhost/43533] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase9.apache.org,36993,1689085214354: Aborting for tests ***** java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:608) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:411) at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.masterFailover(MasterProcedureTestingUtility.java:135) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testRecoveryAndDoubleExecution(TestMasterFailoverWithProcedures.java:314) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testDisableTableWithFailoverAtStep(TestMasterFailoverWithProcedures.java:254) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testDisableTableWithFailover(TestMasterFailoverWithProcedures.java:235) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-11 14:20:20,325 INFO [Listener at localhost/43533] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,36993,1689085214354' ***** 2023-07-11 14:20:20,325 INFO [Listener at localhost/43533] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost/43533 2023-07-11 14:20:20,326 DEBUG [M:0;jenkins-hbase9:36993] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24a64528, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-11 14:20:20,326 INFO [M:0;jenkins-hbase9:36993] regionserver.HRegionServer(1141): aborting server jenkins-hbase9.apache.org,36993,1689085214354 2023-07-11 14:20:20,326 DEBUG [M:0;jenkins-hbase9:36993] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x03e9f0d4 to 127.0.0.1:50917 2023-07-11 14:20:20,327 DEBUG [M:0;jenkins-hbase9:36993] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:20,328 INFO [M:0;jenkins-hbase9:36993] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,36993,1689085214354; all regions closed. 2023-07-11 14:20:20,328 DEBUG [M:0;jenkins-hbase9:36993] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:20,328 DEBUG [M:0;jenkins-hbase9:36993] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-11 14:20:20,328 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-11 14:20:20,328 DEBUG [M:0;jenkins-hbase9:36993] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-11 14:20:20,328 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085216544] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085216544,5,FailOnTimeoutGroup] 2023-07-11 14:20:20,329 INFO [M:0;jenkins-hbase9:36993] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-11 14:20:20,329 INFO [M:0;jenkins-hbase9:36993] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-11 14:20:20,329 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:20,329 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): regionserver:35917-0x10154b3bede0002, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:20,330 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:20,330 INFO [M:0;jenkins-hbase9:36993] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-07-11 14:20:20,328 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085216543] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085216543,5,FailOnTimeoutGroup] 2023-07-11 14:20:20,330 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:20,330 DEBUG [M:0;jenkins-hbase9:36993] master.HMaster(1512): Stopping service threads 2023-07-11 14:20:20,330 INFO [M:0;jenkins-hbase9:36993] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-11 14:20:20,330 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:20,331 ERROR [M:0;jenkins-hbase9:36993] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:20,331 INFO [M:0;jenkins-hbase9:36993] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-11 14:20:20,331 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-11 14:20:20,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35917-0x10154b3bede0002, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:20,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:20,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:20,332 DEBUG [M:0;jenkins-hbase9:36993] zookeeper.ZKUtil(398): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-11 14:20:20,332 WARN [M:0;jenkins-hbase9:36993] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-11 14:20:20,332 INFO [M:0;jenkins-hbase9:36993] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:20,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:20,332 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-11 14:20:20,333 INFO [M:0;jenkins-hbase9:36993] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-11 14:20:20,335 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): regionserver:35917-0x10154b3bede0002, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:20,335 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:20,335 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:20,335 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:20,335 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:20,335 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:20,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:20,336 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,34133,1689085215463 from backup master directory 2023-07-11 14:20:20,341 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:20,341 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,34133,1689085215463 2023-07-11 14:20:20,341 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 14:20:20,341 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:20,341 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase9.apache.org,34133,1689085215463 2023-07-11 14:20:20,348 DEBUG [M:0;jenkins-hbase9:36993] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 14:20:20,348 INFO [M:0;jenkins-hbase9:36993] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:20,349 DEBUG [M:0;jenkins-hbase9:36993] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:20,349 DEBUG [M:0;jenkins-hbase9:36993] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 14:20:20,349 DEBUG [M:0;jenkins-hbase9:36993] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:20,349 ERROR [M:0;jenkins-hbase9:36993] regionserver.HRegion(1825): Memstore data size is 63955 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:20,349 INFO [M:0;jenkins-hbase9:36993] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:20,350 DEBUG [M:0;jenkins-hbase9:36993] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:20,350 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 14:20:20,350 INFO [M:0;jenkins-hbase9:36993] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-11 14:20:20,351 INFO [M:0;jenkins-hbase9:36993] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:36993 2023-07-11 14:20:20,360 DEBUG [M:0;jenkins-hbase9:36993] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,36993,1689085214354 already deleted, retry=false 2023-07-11 14:20:20,367 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:20,408 WARN [RS-EventLoopGroup-4-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:36993 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:36993 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-11 14:20:20,409 DEBUG [RS-EventLoopGroup-4-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:36993 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:36993 2023-07-11 14:20:20,511 DEBUG [RS:0;jenkins-hbase9:35917] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-11 14:20:20,514 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52789, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-11 14:20:20,515 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34133] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:20,532 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:20,532 INFO [M:0;jenkins-hbase9:36993] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,36993,1689085214354; zookeeper connection closed. 2023-07-11 14:20:20,532 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:36993-0x10154b3bede0000, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:20,636 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34133] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:20,739 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34133] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:20,793 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0a72decc to 127.0.0.1:50917 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:20:20,797 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16362918, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:20:20,797 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-11 14:20:20,798 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-11 14:20:20,799 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:20:20,837 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/WALs/jenkins-hbase9.apache.org,36993,1689085214354 to hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/WALs/jenkins-hbase9.apache.org,36993,1689085214354-dead as it is dead 2023-07-11 14:20:20,839 INFO [master/jenkins-hbase9:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/WALs/jenkins-hbase9.apache.org,36993,1689085214354-dead/jenkins-hbase9.apache.org%2C36993%2C1689085214354.1689085215952 2023-07-11 14:20:20,842 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34133] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:20,843 INFO [master/jenkins-hbase9:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/WALs/jenkins-hbase9.apache.org,36993,1689085214354-dead/jenkins-hbase9.apache.org%2C36993%2C1689085214354.1689085215952 after 4ms 2023-07-11 14:20:20,844 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/WALs/jenkins-hbase9.apache.org,36993,1689085214354-dead/jenkins-hbase9.apache.org%2C36993%2C1689085214354.1689085215952 to hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C36993%2C1689085214354.1689085215952 2023-07-11 14:20:20,844 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/WALs/jenkins-hbase9.apache.org,36993,1689085214354-dead 2023-07-11 14:20:20,845 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/WALs/jenkins-hbase9.apache.org,34133,1689085215463 2023-07-11 14:20:20,848 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C34133%2C1689085215463, suffix=, logDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/WALs/jenkins-hbase9.apache.org,34133,1689085215463, archiveDir=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/oldWALs, maxLogs=10 2023-07-11 14:20:20,861 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45455,DS-960c0af4-f4f5-4bac-8f7c-1e8a57a16cc2,DISK] 2023-07-11 14:20:20,865 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/WALs/jenkins-hbase9.apache.org,34133,1689085215463/jenkins-hbase9.apache.org%2C34133%2C1689085215463.1689085220848 2023-07-11 14:20:20,865 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45455,DS-960c0af4-f4f5-4bac-8f7c-1e8a57a16cc2,DISK]] 2023-07-11 14:20:20,865 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:20,865 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:20,865 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:20,865 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:20,867 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:20,868 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-11 14:20:20,869 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-11 14:20:20,870 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:20,870 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-11 14:20:20,871 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C36993%2C1689085214354.1689085215952 2023-07-11 14:20:20,894 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 156, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=134, path=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C36993%2C1689085214354.1689085215952 2023-07-11 14:20:20,895 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=62.46 KB heapSize=74.24 KB 2023-07-11 14:20:20,947 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34133] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:21,050 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34133] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:21,153 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34133] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:21,255 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34133] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:21,348 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=62.46 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/18df01ab79e349bc9fff251eb3615cc1 2023-07-11 14:20:21,358 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34133] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:21,389 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/18df01ab79e349bc9fff251eb3615cc1 as hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/18df01ab79e349bc9fff251eb3615cc1 2023-07-11 14:20:21,398 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/18df01ab79e349bc9fff251eb3615cc1, entries=156, sequenceid=134, filesize=67.4 K 2023-07-11 14:20:21,400 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~62.46 KB/63955, heapSize ~74.23 KB/76008, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 505ms, sequenceid=134, compaction requested=false; wal=null 2023-07-11 14:20:21,401 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C36993%2C1689085214354.1689085215952 2023-07-11 14:20:21,405 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:21,409 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/134.seqid, newMaxSeqId=134, maxSeqId=1 2023-07-11 14:20:21,410 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=135; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72852264, jitterRate=0.08558332920074463}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 14:20:21,410 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:21,411 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-11 14:20:21,412 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:21,412 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:21,412 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:21,413 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:21,425 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] store.ProcedureTree(114): Procedure Procedure(pid=18, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DisableTableProcedure) stack ids=[0, 1, 2] 2023-07-11 14:20:21,425 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:21,425 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:21,426 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:21,426 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:21,427 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:21,427 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(468): Loading pid=18, state=RUNNABLE:DISABLE_TABLE_MARK_REGIONS_OFFLINE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:21,427 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.Procedure(966): pid=18, state=RUNNABLE:DISABLE_TABLE_MARK_REGIONS_OFFLINE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:21,427 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 14 msec 2023-07-11 14:20:21,427 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-11 14:20:21,428 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:21,429 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,35917,1689085215520, table=hbase:meta, region=1588230740 2023-07-11 14:20:21,430 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 1 possibly 'live' servers, and 0 'splitting'. 2023-07-11 14:20:21,431 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:21,442 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-11 14:20:21,442 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-11 14:20:21,442 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-11 14:20:21,443 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-11 14:20:21,443 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-11 14:20:21,444 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-11 14:20:21,444 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-11 14:20:21,446 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-11 14:20:21,446 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase9.apache.org,34133,1689085215463, sessionid=0x10154b3bede0001, setting cluster-up flag (Was=true) 2023-07-11 14:20:21,448 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-11 14:20:21,450 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,34133,1689085215463 2023-07-11 14:20:21,452 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-11 14:20:21,454 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,34133,1689085215463 2023-07-11 14:20:21,454 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/.hbase-snapshot/.tmp 2023-07-11 14:20:21,456 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:21,456 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:21,456 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:21,456 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:21,456 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-07-11 14:20:21,456 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:21,456 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-11 14:20:21,456 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:21,457 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085251457 2023-07-11 14:20:21,457 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-11 14:20:21,458 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:DISABLE_TABLE_MARK_REGIONS_OFFLINE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-11 14:20:21,458 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-11 14:20:21,458 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-11 14:20:21,458 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-11 14:20:21,458 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-11 14:20:21,458 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-11 14:20:21,458 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:21,459 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-11 14:20:21,459 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-11 14:20:21,459 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-11 14:20:21,459 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-11 14:20:21,460 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-11 14:20:21,460 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085221460,5,FailOnTimeoutGroup] 2023-07-11 14:20:21,460 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085221460,5,FailOnTimeoutGroup] 2023-07-11 14:20:21,460 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:21,460 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-11 14:20:21,460 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:21,460 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:21,460 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689085221460, completionTime=-1 2023-07-11 14:20:21,460 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-11 14:20:21,460 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:21,463 DEBUG [hconnection-0x4e060fb1-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 14:20:21,465 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:33094, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 14:20:21,468 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=a48689024a2a1f1a7a4accb24d7943d7, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:21,468 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=7678ac0cab253467f45346026a631b20, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:21,468 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=9901b6faf1d3748db88cfd3a05b4468a, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:21,469 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=74c58b571981d83dcf418f1080d591b9, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:21,469 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=cdb24378120768c854e74a1b1338ae63, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,35917,1689085215520, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520, openSeqNum=2 2023-07-11 14:20:21,469 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:21,469 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085281469 2023-07-11 14:20:21,469 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085341469 2023-07-11 14:20:21,469 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 9 msec 2023-07-11 14:20:21,488 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,34133,1689085215463-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:21,488 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,34133,1689085215463-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:21,488 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,34133,1689085215463-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:21,488 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:34133, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:21,488 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:21,489 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-11 14:20:21,495 DEBUG [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-11 14:20:21,507 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-11 14:20:21,510 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-11 14:20:21,511 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 1.168sec 2023-07-11 14:20:21,511 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-11 14:20:21,511 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-11 14:20:21,511 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-11 14:20:21,511 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,34133,1689085215463-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-11 14:20:21,511 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7678ac0cab253467f45346026a631b20, UNASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=9901b6faf1d3748db88cfd3a05b4468a, UNASSIGN}, {pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=74c58b571981d83dcf418f1080d591b9, UNASSIGN}, {pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=cdb24378120768c854e74a1b1338ae63, UNASSIGN}] 2023-07-11 14:20:21,511 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,34133,1689085215463-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-11 14:20:21,522 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=cdb24378120768c854e74a1b1338ae63, UNASSIGN 2023-07-11 14:20:21,525 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=cdb24378120768c854e74a1b1338ae63, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:21,525 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085221525"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085221525"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085221525"}]},"ts":"1689085221525"} 2023-07-11 14:20:21,529 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; CloseRegionProcedure cdb24378120768c854e74a1b1338ae63, server=jenkins-hbase9.apache.org,35917,1689085215520}] 2023-07-11 14:20:21,545 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=74c58b571981d83dcf418f1080d591b9, UNASSIGN 2023-07-11 14:20:21,546 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=74c58b571981d83dcf418f1080d591b9, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:21,547 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689085221546"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085221546"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085221546"}]},"ts":"1689085221546"} 2023-07-11 14:20:21,547 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-11 14:20:21,550 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=21, state=RUNNABLE; CloseRegionProcedure 74c58b571981d83dcf418f1080d591b9, server=jenkins-hbase9.apache.org,35917,1689085215520}] 2023-07-11 14:20:21,554 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=9901b6faf1d3748db88cfd3a05b4468a, UNASSIGN 2023-07-11 14:20:21,555 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=9901b6faf1d3748db88cfd3a05b4468a, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:21,555 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689085221555"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085221555"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085221555"}]},"ts":"1689085221555"} 2023-07-11 14:20:21,557 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=20, state=RUNNABLE; CloseRegionProcedure 9901b6faf1d3748db88cfd3a05b4468a, server=jenkins-hbase9.apache.org,35917,1689085215520}] 2023-07-11 14:20:21,561 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7678ac0cab253467f45346026a631b20, UNASSIGN 2023-07-11 14:20:21,562 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=7678ac0cab253467f45346026a631b20, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:21,562 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085221562"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085221562"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085221562"}]},"ts":"1689085221562"} 2023-07-11 14:20:21,564 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=19, state=RUNNABLE; CloseRegionProcedure 7678ac0cab253467f45346026a631b20, server=jenkins-hbase9.apache.org,35917,1689085215520}] 2023-07-11 14:20:21,688 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:21,688 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-11 14:20:21,691 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:33104, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-11 14:20:21,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(111): Close 74c58b571981d83dcf418f1080d591b9 2023-07-11 14:20:21,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 74c58b571981d83dcf418f1080d591b9, disabling compactions & flushes 2023-07-11 14:20:21,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9. 2023-07-11 14:20:21,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9. 2023-07-11 14:20:21,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9. after waiting 0 ms 2023-07-11 14:20:21,694 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9. 2023-07-11 14:20:21,700 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/74c58b571981d83dcf418f1080d591b9/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-11 14:20:21,701 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9. 2023-07-11 14:20:21,701 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 74c58b571981d83dcf418f1080d591b9: 2023-07-11 14:20:21,705 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(149): Closed 74c58b571981d83dcf418f1080d591b9 2023-07-11 14:20:21,705 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(111): Close 7678ac0cab253467f45346026a631b20 2023-07-11 14:20:21,705 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 7678ac0cab253467f45346026a631b20, disabling compactions & flushes 2023-07-11 14:20:21,705 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20. 2023-07-11 14:20:21,705 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20. 2023-07-11 14:20:21,705 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20. after waiting 0 ms 2023-07-11 14:20:21,705 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20. 2023-07-11 14:20:21,706 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=74c58b571981d83dcf418f1080d591b9, regionState=CLOSED 2023-07-11 14:20:21,706 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,b,1689085218009.74c58b571981d83dcf418f1080d591b9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689085221705"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085221705"}]},"ts":"1689085221705"} 2023-07-11 14:20:21,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/7678ac0cab253467f45346026a631b20/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-11 14:20:21,712 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=21 2023-07-11 14:20:21,712 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=21, state=SUCCESS; CloseRegionProcedure 74c58b571981d83dcf418f1080d591b9, server=jenkins-hbase9.apache.org,35917,1689085215520 in 158 msec 2023-07-11 14:20:21,713 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20. 2023-07-11 14:20:21,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 7678ac0cab253467f45346026a631b20: 2023-07-11 14:20:21,714 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=74c58b571981d83dcf418f1080d591b9, UNASSIGN in 201 msec 2023-07-11 14:20:21,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(149): Closed 7678ac0cab253467f45346026a631b20 2023-07-11 14:20:21,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(111): Close cdb24378120768c854e74a1b1338ae63 2023-07-11 14:20:21,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing cdb24378120768c854e74a1b1338ae63, disabling compactions & flushes 2023-07-11 14:20:21,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63. 2023-07-11 14:20:21,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63. 2023-07-11 14:20:21,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63. after waiting 0 ms 2023-07-11 14:20:21,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63. 2023-07-11 14:20:21,716 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=7678ac0cab253467f45346026a631b20, regionState=CLOSED 2023-07-11 14:20:21,716 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,,1689085218009.7678ac0cab253467f45346026a631b20.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085221716"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085221716"}]},"ts":"1689085221716"} 2023-07-11 14:20:21,721 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=19 2023-07-11 14:20:21,721 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=19, state=SUCCESS; CloseRegionProcedure 7678ac0cab253467f45346026a631b20, server=jenkins-hbase9.apache.org,35917,1689085215520 in 155 msec 2023-07-11 14:20:21,722 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/cdb24378120768c854e74a1b1338ae63/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-11 14:20:21,723 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=7678ac0cab253467f45346026a631b20, UNASSIGN in 211 msec 2023-07-11 14:20:21,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63. 2023-07-11 14:20:21,724 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for cdb24378120768c854e74a1b1338ae63: 2023-07-11 14:20:21,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(149): Closed cdb24378120768c854e74a1b1338ae63 2023-07-11 14:20:21,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(111): Close 9901b6faf1d3748db88cfd3a05b4468a 2023-07-11 14:20:21,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 9901b6faf1d3748db88cfd3a05b4468a, disabling compactions & flushes 2023-07-11 14:20:21,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a. 2023-07-11 14:20:21,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a. 2023-07-11 14:20:21,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a. after waiting 0 ms 2023-07-11 14:20:21,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a. 2023-07-11 14:20:21,727 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=cdb24378120768c854e74a1b1338ae63, regionState=CLOSED 2023-07-11 14:20:21,728 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,c,1689085218009.cdb24378120768c854e74a1b1338ae63.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085221727"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085221727"}]},"ts":"1689085221727"} 2023-07-11 14:20:21,732 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-11 14:20:21,733 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; CloseRegionProcedure cdb24378120768c854e74a1b1338ae63, server=jenkins-hbase9.apache.org,35917,1689085215520 in 201 msec 2023-07-11 14:20:21,733 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/default/testDisableTableWithFailoverAtStep3/9901b6faf1d3748db88cfd3a05b4468a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-11 14:20:21,734 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a. 2023-07-11 14:20:21,734 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 9901b6faf1d3748db88cfd3a05b4468a: 2023-07-11 14:20:21,735 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=cdb24378120768c854e74a1b1338ae63, UNASSIGN in 222 msec 2023-07-11 14:20:21,737 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(149): Closed 9901b6faf1d3748db88cfd3a05b4468a 2023-07-11 14:20:21,737 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=9901b6faf1d3748db88cfd3a05b4468a, regionState=CLOSED 2023-07-11 14:20:21,738 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,a,1689085218009.9901b6faf1d3748db88cfd3a05b4468a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689085221737"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085221737"}]},"ts":"1689085221737"} 2023-07-11 14:20:21,743 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=20 2023-07-11 14:20:21,743 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=20, state=SUCCESS; CloseRegionProcedure 9901b6faf1d3748db88cfd3a05b4468a, server=jenkins-hbase9.apache.org,35917,1689085215520 in 183 msec 2023-07-11 14:20:21,745 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=18 2023-07-11 14:20:21,746 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=9901b6faf1d3748db88cfd3a05b4468a, UNASSIGN in 232 msec 2023-07-11 14:20:21,747 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDisableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085221747"}]},"ts":"1689085221747"} 2023-07-11 14:20:21,749 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDisableTableWithFailoverAtStep3, state=DISABLED in hbase:meta 2023-07-11 14:20:21,751 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testDisableTableWithFailoverAtStep3 to state=DISABLED 2023-07-11 14:20:21,754 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 in 2.4890 sec 2023-07-11 14:20:21,830 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-11 14:20:21,831 DEBUG [Listener at localhost/43533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07e87950 to 127.0.0.1:50917 2023-07-11 14:20:21,831 DEBUG [Listener at localhost/43533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:21,832 DEBUG [Listener at localhost/43533] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-11 14:20:21,832 DEBUG [Listener at localhost/43533] util.JVMClusterUtil(257): Found active master hash=612143531, stopped=false 2023-07-11 14:20:21,832 INFO [Listener at localhost/43533] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase9.apache.org,34133,1689085215463 2023-07-11 14:20:21,834 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:21,834 INFO [Listener at localhost/43533] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:21,834 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:21,834 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): regionserver:35917-0x10154b3bede0002, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:21,835 DEBUG [Listener at localhost/43533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a72decc to 127.0.0.1:50917 2023-07-11 14:20:21,835 DEBUG [Listener at localhost/43533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:21,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:21,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35917-0x10154b3bede0002, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:21,835 INFO [Listener at localhost/43533] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,35917,1689085215520' ***** 2023-07-11 14:20:21,836 INFO [Listener at localhost/43533] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-11 14:20:21,836 INFO [RS:0;jenkins-hbase9:35917] regionserver.HeapMemoryManager(220): Stopping 2023-07-11 14:20:21,836 INFO [RS:0;jenkins-hbase9:35917] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-11 14:20:21,836 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-11 14:20:21,836 INFO [RS:0;jenkins-hbase9:35917] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-11 14:20:21,837 INFO [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer(3305): Received CLOSE for a48689024a2a1f1a7a4accb24d7943d7 2023-07-11 14:20:21,837 INFO [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer(1144): stopping server jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:21,838 DEBUG [RS:0;jenkins-hbase9:35917] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2dda002c to 127.0.0.1:50917 2023-07-11 14:20:21,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing a48689024a2a1f1a7a4accb24d7943d7, disabling compactions & flushes 2023-07-11 14:20:21,838 DEBUG [RS:0;jenkins-hbase9:35917] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:21,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7. 2023-07-11 14:20:21,838 INFO [RS:0;jenkins-hbase9:35917] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-11 14:20:21,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7. 2023-07-11 14:20:21,838 INFO [RS:0;jenkins-hbase9:35917] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-11 14:20:21,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7. after waiting 0 ms 2023-07-11 14:20:21,838 INFO [RS:0;jenkins-hbase9:35917] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-11 14:20:21,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7. 2023-07-11 14:20:21,839 INFO [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-11 14:20:21,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing a48689024a2a1f1a7a4accb24d7943d7 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-11 14:20:21,839 INFO [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-07-11 14:20:21,839 DEBUG [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, a48689024a2a1f1a7a4accb24d7943d7=hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7.} 2023-07-11 14:20:21,842 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-11 14:20:21,842 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-11 14:20:21,842 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-11 14:20:21,842 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-11 14:20:21,842 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-11 14:20:21,842 DEBUG [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer(1504): Waiting on 1588230740, a48689024a2a1f1a7a4accb24d7943d7 2023-07-11 14:20:21,842 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=10.44 KB heapSize=16.86 KB 2023-07-11 14:20:21,874 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=10.02 KB at sequenceid=31 (bloomFilter=false), to=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/.tmp/info/6dbf3662fdd047628ab3a4bc243fa543 2023-07-11 14:20:21,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/namespace/a48689024a2a1f1a7a4accb24d7943d7/.tmp/info/0a9358ac9c234956b018d5ff8e5e58e5 2023-07-11 14:20:21,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/namespace/a48689024a2a1f1a7a4accb24d7943d7/.tmp/info/0a9358ac9c234956b018d5ff8e5e58e5 as hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/namespace/a48689024a2a1f1a7a4accb24d7943d7/info/0a9358ac9c234956b018d5ff8e5e58e5 2023-07-11 14:20:21,898 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=429 B at sequenceid=31 (bloomFilter=false), to=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/.tmp/table/b55d2a30041547e999ddaef2033b6b11 2023-07-11 14:20:21,899 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/namespace/a48689024a2a1f1a7a4accb24d7943d7/info/0a9358ac9c234956b018d5ff8e5e58e5, entries=2, sequenceid=6, filesize=4.8 K 2023-07-11 14:20:21,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for a48689024a2a1f1a7a4accb24d7943d7 in 62ms, sequenceid=6, compaction requested=false 2023-07-11 14:20:21,903 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-11 14:20:21,908 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/.tmp/info/6dbf3662fdd047628ab3a4bc243fa543 as hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/info/6dbf3662fdd047628ab3a4bc243fa543 2023-07-11 14:20:21,912 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/namespace/a48689024a2a1f1a7a4accb24d7943d7/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-11 14:20:21,913 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7. 2023-07-11 14:20:21,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for a48689024a2a1f1a7a4accb24d7943d7: 2023-07-11 14:20:21,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689085217268.a48689024a2a1f1a7a4accb24d7943d7. 2023-07-11 14:20:21,915 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/info/6dbf3662fdd047628ab3a4bc243fa543, entries=54, sequenceid=31, filesize=12.4 K 2023-07-11 14:20:21,917 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/.tmp/table/b55d2a30041547e999ddaef2033b6b11 as hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/table/b55d2a30041547e999ddaef2033b6b11 2023-07-11 14:20:21,924 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/table/b55d2a30041547e999ddaef2033b6b11, entries=5, sequenceid=31, filesize=4.9 K 2023-07-11 14:20:21,926 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~10.44 KB/10692, heapSize ~16.58 KB/16976, currentSize=0 B/0 for 1588230740 in 83ms, sequenceid=31, compaction requested=false 2023-07-11 14:20:21,926 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-11 14:20:21,929 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-11 14:20:21,932 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/data/hbase/meta/1588230740/recovered.edits/34.seqid, newMaxSeqId=34, maxSeqId=1 2023-07-11 14:20:21,933 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-11 14:20:21,934 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-11 14:20:21,934 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-11 14:20:21,934 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-11 14:20:22,042 INFO [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,35917,1689085215520; all regions closed. 2023-07-11 14:20:22,052 DEBUG [RS:0;jenkins-hbase9:35917] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/oldWALs 2023-07-11 14:20:22,052 INFO [RS:0;jenkins-hbase9:35917] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C35917%2C1689085215520.meta:.meta(num 1689085217018) 2023-07-11 14:20:22,058 DEBUG [RS:0;jenkins-hbase9:35917] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/oldWALs 2023-07-11 14:20:22,058 INFO [RS:0;jenkins-hbase9:35917] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C35917%2C1689085215520:(num 1689085216855) 2023-07-11 14:20:22,058 DEBUG [RS:0;jenkins-hbase9:35917] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:22,058 INFO [RS:0;jenkins-hbase9:35917] regionserver.LeaseManager(133): Closed leases 2023-07-11 14:20:22,058 INFO [RS:0;jenkins-hbase9:35917] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-11 14:20:22,058 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 14:20:22,059 INFO [RS:0;jenkins-hbase9:35917] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:35917 2023-07-11 14:20:22,064 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): regionserver:35917-0x10154b3bede0002, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,35917,1689085215520 2023-07-11 14:20:22,064 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 14:20:22,064 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): regionserver:35917-0x10154b3bede0002, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 14:20:22,072 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,35917,1689085215520] 2023-07-11 14:20:22,072 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,35917,1689085215520; numProcessing=1 2023-07-11 14:20:22,074 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,35917,1689085215520 already deleted, retry=false 2023-07-11 14:20:22,074 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,35917,1689085215520 expired; onlineServers=0 2023-07-11 14:20:22,074 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,34133,1689085215463' ***** 2023-07-11 14:20:22,074 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-11 14:20:22,074 DEBUG [M:1;jenkins-hbase9:34133] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65419b90, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-11 14:20:22,074 INFO [M:1;jenkins-hbase9:34133] regionserver.HRegionServer(1144): stopping server jenkins-hbase9.apache.org,34133,1689085215463 2023-07-11 14:20:22,074 INFO [M:1;jenkins-hbase9:34133] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,34133,1689085215463; all regions closed. 2023-07-11 14:20:22,074 DEBUG [M:1;jenkins-hbase9:34133] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:22,074 DEBUG [M:1;jenkins-hbase9:34133] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-11 14:20:22,074 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-11 14:20:22,074 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085221460] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085221460,5,FailOnTimeoutGroup] 2023-07-11 14:20:22,074 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085221460] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085221460,5,FailOnTimeoutGroup] 2023-07-11 14:20:22,074 DEBUG [M:1;jenkins-hbase9:34133] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-11 14:20:22,075 INFO [M:1;jenkins-hbase9:34133] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-11 14:20:22,076 INFO [M:1;jenkins-hbase9:34133] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-11 14:20:22,076 INFO [M:1;jenkins-hbase9:34133] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-07-11 14:20:22,076 DEBUG [M:1;jenkins-hbase9:34133] master.HMaster(1512): Stopping service threads 2023-07-11 14:20:22,076 INFO [M:1;jenkins-hbase9:34133] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-11 14:20:22,076 ERROR [M:1;jenkins-hbase9:34133] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:22,076 INFO [M:1;jenkins-hbase9:34133] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:22,076 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-11 14:20:22,076 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:22,076 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:22,077 DEBUG [M:1;jenkins-hbase9:34133] zookeeper.ZKUtil(398): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-11 14:20:22,077 WARN [M:1;jenkins-hbase9:34133] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-11 14:20:22,077 INFO [M:1;jenkins-hbase9:34133] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:22,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:22,077 INFO [M:1;jenkins-hbase9:34133] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-11 14:20:22,078 DEBUG [M:1;jenkins-hbase9:34133] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 14:20:22,078 INFO [M:1;jenkins-hbase9:34133] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:22,078 DEBUG [M:1;jenkins-hbase9:34133] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:22,078 DEBUG [M:1;jenkins-hbase9:34133] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 14:20:22,078 DEBUG [M:1;jenkins-hbase9:34133] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:22,078 INFO [M:1;jenkins-hbase9:34133] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=23.35 KB heapSize=28.81 KB 2023-07-11 14:20:22,233 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): regionserver:35917-0x10154b3bede0002, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:22,234 INFO [RS:0;jenkins-hbase9:35917] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,35917,1689085215520; zookeeper connection closed. 2023-07-11 14:20:22,234 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): regionserver:35917-0x10154b3bede0002, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:22,234 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@60e63b8a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@60e63b8a 2023-07-11 14:20:22,235 INFO [Listener at localhost/43533] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 1 regionserver(s) complete 2023-07-11 14:20:22,495 INFO [M:1;jenkins-hbase9:34133] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=23.35 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f0306d4efe254505a7ccf5cbf8f6cbf9 2023-07-11 14:20:22,502 DEBUG [M:1;jenkins-hbase9:34133] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f0306d4efe254505a7ccf5cbf8f6cbf9 as hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f0306d4efe254505a7ccf5cbf8f6cbf9 2023-07-11 14:20:22,508 INFO [M:1;jenkins-hbase9:34133] regionserver.HStore(1080): Added hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f0306d4efe254505a7ccf5cbf8f6cbf9, entries=9, sequenceid=190, filesize=5.4 K 2023-07-11 14:20:22,510 INFO [M:1;jenkins-hbase9:34133] regionserver.HRegion(2948): Finished flush of dataSize ~23.35 KB/23913, heapSize ~28.80 KB/29488, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 431ms, sequenceid=190, compaction requested=false 2023-07-11 14:20:22,511 INFO [M:1;jenkins-hbase9:34133] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:22,511 DEBUG [M:1;jenkins-hbase9:34133] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:22,515 INFO [M:1;jenkins-hbase9:34133] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-11 14:20:22,515 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 14:20:22,515 INFO [M:1;jenkins-hbase9:34133] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:34133 2023-07-11 14:20:22,518 DEBUG [M:1;jenkins-hbase9:34133] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,34133,1689085215463 already deleted, retry=false 2023-07-11 14:20:22,573 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-11 14:20:22,620 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:22,620 INFO [M:1;jenkins-hbase9:34133] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,34133,1689085215463; zookeeper connection closed. 2023-07-11 14:20:22,620 DEBUG [Listener at localhost/43533-EventThread] zookeeper.ZKWatcher(600): master:34133-0x10154b3bede0001, quorum=127.0.0.1:50917, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:22,621 WARN [Listener at localhost/43533] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-11 14:20:22,627 INFO [Listener at localhost/43533] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-11 14:20:22,730 WARN [BP-1708462836-172.31.2.10-1689085211229 heartbeating to localhost/127.0.0.1:46725] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-11 14:20:22,730 WARN [BP-1708462836-172.31.2.10-1689085211229 heartbeating to localhost/127.0.0.1:46725] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1708462836-172.31.2.10-1689085211229 (Datanode Uuid b127a362-8312-4dd6-bceb-e37ed2152684) service to localhost/127.0.0.1:46725 2023-07-11 14:20:22,732 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/cluster_acd48a5f-1519-3a78-5e80-8455ee69e37a/dfs/data/data1/current/BP-1708462836-172.31.2.10-1689085211229] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-11 14:20:22,732 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/cluster_acd48a5f-1519-3a78-5e80-8455ee69e37a/dfs/data/data2/current/BP-1708462836-172.31.2.10-1689085211229] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-11 14:20:22,760 INFO [Listener at localhost/43533] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-11 14:20:22,871 INFO [Listener at localhost/43533] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-11 14:20:22,897 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-07-11 14:20:22,909 INFO [Listener at localhost/43533] hbase.ResourceChecker(175): after: master.procedure.TestMasterFailoverWithProcedures#testDisableTableWithFailover Thread=62 (was 10) Potentially hanging thread: nioEventLoopGroup-2-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/43533 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Idle-Rpc-Conn-Sweeper-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase9:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:46725 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-3-worker-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: IPC Parameter Sending Thread #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-3-worker-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase9:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase9:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@284da65d java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:327) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-3-worker-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: RS-EventLoopGroup-1-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (200715596) connection to localhost/127.0.0.1:46725 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-2-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SessionTracker java.lang.Thread.sleep(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-3-worker-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: IPC Client (200715596) connection to localhost/127.0.0.1:46725 from jenkins.hfs.0 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: IPC Client (200715596) connection to localhost/127.0.0.1:46725 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RpcClient-timer-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.0@localhost:46725 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-1 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3693) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-3-worker-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: nioEventLoopGroup-3-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=394 (was 263) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=333 (was 385), ProcessCount=171 (was 171), AvailableMemoryMB=7748 (was 8318) 2023-07-11 14:20:22,918 INFO [Listener at localhost/43533] hbase.ResourceChecker(147): before: master.procedure.TestMasterFailoverWithProcedures#testCreateWithFailover Thread=62, OpenFileDescriptor=394, MaxFileDescriptor=60000, SystemLoadAverage=333, ProcessCount=171, AvailableMemoryMB=7749 2023-07-11 14:20:22,918 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=2, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-11 14:20:22,918 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/hadoop.log.dir so I do NOT create it in target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09 2023-07-11 14:20:22,918 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/898f6d93-c3ce-f1d0-bc44-42c3cdf7f832/hadoop.tmp.dir so I do NOT create it in target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09 2023-07-11 14:20:22,918 INFO [Listener at localhost/43533] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/cluster_e8c9e4a3-15af-961f-cdce-f726bad055a8, deleteOnExit=true 2023-07-11 14:20:22,918 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-11 14:20:22,919 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/test.cache.data in system properties and HBase conf 2023-07-11 14:20:22,919 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/hadoop.tmp.dir in system properties and HBase conf 2023-07-11 14:20:22,919 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/hadoop.log.dir in system properties and HBase conf 2023-07-11 14:20:22,919 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-11 14:20:22,919 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-11 14:20:22,919 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-11 14:20:22,919 DEBUG [Listener at localhost/43533] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-11 14:20:22,919 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-11 14:20:22,920 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-11 14:20:22,920 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-11 14:20:22,920 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-11 14:20:22,920 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-11 14:20:22,920 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-11 14:20:22,920 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-11 14:20:22,920 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-11 14:20:22,920 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-11 14:20:22,920 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/nfs.dump.dir in system properties and HBase conf 2023-07-11 14:20:22,921 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/java.io.tmpdir in system properties and HBase conf 2023-07-11 14:20:22,921 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-11 14:20:22,921 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-11 14:20:22,921 INFO [Listener at localhost/43533] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-11 14:20:22,925 WARN [Listener at localhost/43533] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-11 14:20:22,926 WARN [Listener at localhost/43533] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-11 14:20:22,975 WARN [Listener at localhost/43533] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-11 14:20:22,978 INFO [Listener at localhost/43533] log.Slf4jLog(67): jetty-6.1.26 2023-07-11 14:20:22,983 INFO [Listener at localhost/43533] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/java.io.tmpdir/Jetty_localhost_40095_hdfs____7wmqre/webapp 2023-07-11 14:20:23,080 INFO [Listener at localhost/43533] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40095 2023-07-11 14:20:23,085 WARN [Listener at localhost/43533] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-11 14:20:23,085 WARN [Listener at localhost/43533] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-11 14:20:23,140 WARN [Listener at localhost/44189] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-11 14:20:23,157 WARN [Listener at localhost/44189] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-11 14:20:23,160 WARN [Listener at localhost/44189] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-11 14:20:23,161 INFO [Listener at localhost/44189] log.Slf4jLog(67): jetty-6.1.26 2023-07-11 14:20:23,165 INFO [Listener at localhost/44189] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/java.io.tmpdir/Jetty_localhost_46137_datanode____.lu6fa/webapp 2023-07-11 14:20:23,274 INFO [Listener at localhost/44189] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46137 2023-07-11 14:20:23,281 WARN [Listener at localhost/43015] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-11 14:20:23,381 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xaa89e1454ce59ede: Processing first storage report for DS-12462c9e-c2e2-4c0d-b8b4-7a67c6de1261 from datanode c08d77ce-3bb6-49a3-9818-1a102cf00a32 2023-07-11 14:20:23,382 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xaa89e1454ce59ede: from storage DS-12462c9e-c2e2-4c0d-b8b4-7a67c6de1261 node DatanodeRegistration(127.0.0.1:43635, datanodeUuid=c08d77ce-3bb6-49a3-9818-1a102cf00a32, infoPort=46301, infoSecurePort=0, ipcPort=43015, storageInfo=lv=-57;cid=testClusterID;nsid=2039651001;c=1689085222928), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-11 14:20:23,382 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xaa89e1454ce59ede: Processing first storage report for DS-f6dc23b7-6dde-412a-bb8b-9d932a6fa836 from datanode c08d77ce-3bb6-49a3-9818-1a102cf00a32 2023-07-11 14:20:23,382 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xaa89e1454ce59ede: from storage DS-f6dc23b7-6dde-412a-bb8b-9d932a6fa836 node DatanodeRegistration(127.0.0.1:43635, datanodeUuid=c08d77ce-3bb6-49a3-9818-1a102cf00a32, infoPort=46301, infoSecurePort=0, ipcPort=43015, storageInfo=lv=-57;cid=testClusterID;nsid=2039651001;c=1689085222928), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-11 14:20:23,391 DEBUG [Listener at localhost/43015] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09 2023-07-11 14:20:23,393 INFO [Listener at localhost/43015] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/cluster_e8c9e4a3-15af-961f-cdce-f726bad055a8/zookeeper_0, clientPort=60113, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/cluster_e8c9e4a3-15af-961f-cdce-f726bad055a8/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/cluster_e8c9e4a3-15af-961f-cdce-f726bad055a8/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-11 14:20:23,395 INFO [Listener at localhost/43015] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=60113 2023-07-11 14:20:23,395 INFO [Listener at localhost/43015] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:23,396 INFO [Listener at localhost/43015] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:23,813 INFO [Listener at localhost/43015] util.FSUtils(471): Created version file at hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d with version=8 2023-07-11 14:20:23,813 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/hbase-staging 2023-07-11 14:20:23,815 INFO [Listener at localhost/43015] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-07-11 14:20:23,815 INFO [Listener at localhost/43015] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:23,815 INFO [Listener at localhost/43015] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:23,816 INFO [Listener at localhost/43015] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 14:20:23,816 INFO [Listener at localhost/43015] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:23,816 INFO [Listener at localhost/43015] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 14:20:23,816 INFO [Listener at localhost/43015] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 14:20:23,817 INFO [Listener at localhost/43015] ipc.NettyRpcServer(120): Bind to /172.31.2.10:43201 2023-07-11 14:20:23,817 INFO [Listener at localhost/43015] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:23,819 INFO [Listener at localhost/43015] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:23,820 INFO [Listener at localhost/43015] zookeeper.RecoverableZooKeeper(93): Process identifier=master:43201 connecting to ZooKeeper ensemble=127.0.0.1:60113 2023-07-11 14:20:23,826 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:432010x0, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 14:20:23,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:43201-0x10154b3e5800000 connected 2023-07-11 14:20:23,842 DEBUG [Listener at localhost/43015] zookeeper.ZKUtil(164): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:23,842 DEBUG [Listener at localhost/43015] zookeeper.ZKUtil(164): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:23,842 DEBUG [Listener at localhost/43015] zookeeper.ZKUtil(164): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 14:20:23,843 DEBUG [Listener at localhost/43015] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43201 2023-07-11 14:20:23,843 DEBUG [Listener at localhost/43015] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43201 2023-07-11 14:20:23,843 DEBUG [Listener at localhost/43015] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43201 2023-07-11 14:20:23,843 DEBUG [Listener at localhost/43015] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43201 2023-07-11 14:20:23,844 DEBUG [Listener at localhost/43015] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43201 2023-07-11 14:20:23,844 INFO [Listener at localhost/43015] master.HMaster(444): hbase.rootdir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d, hbase.cluster.distributed=false 2023-07-11 14:20:23,846 INFO [Listener at localhost/43015] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-07-11 14:20:23,847 INFO [Listener at localhost/43015] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:23,847 INFO [Listener at localhost/43015] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:23,847 INFO [Listener at localhost/43015] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 14:20:23,847 INFO [Listener at localhost/43015] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:23,847 INFO [Listener at localhost/43015] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 14:20:23,847 INFO [Listener at localhost/43015] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 14:20:23,848 INFO [Listener at localhost/43015] ipc.NettyRpcServer(120): Bind to /172.31.2.10:36779 2023-07-11 14:20:23,849 INFO [Listener at localhost/43015] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:23,850 INFO [Listener at localhost/43015] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:23,851 INFO [Listener at localhost/43015] zookeeper.RecoverableZooKeeper(93): Process identifier=master:36779 connecting to ZooKeeper ensemble=127.0.0.1:60113 2023-07-11 14:20:23,853 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:367790x0, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 14:20:23,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:36779-0x10154b3e5800001 connected 2023-07-11 14:20:23,857 DEBUG [Listener at localhost/43015] zookeeper.ZKUtil(164): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:23,857 DEBUG [Listener at localhost/43015] zookeeper.ZKUtil(164): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:23,858 DEBUG [Listener at localhost/43015] zookeeper.ZKUtil(164): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 14:20:23,859 DEBUG [Listener at localhost/43015] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36779 2023-07-11 14:20:23,859 DEBUG [Listener at localhost/43015] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36779 2023-07-11 14:20:23,859 DEBUG [Listener at localhost/43015] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36779 2023-07-11 14:20:23,860 DEBUG [Listener at localhost/43015] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36779 2023-07-11 14:20:23,860 DEBUG [Listener at localhost/43015] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36779 2023-07-11 14:20:23,860 INFO [Listener at localhost/43015] master.HMaster(444): hbase.rootdir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d, hbase.cluster.distributed=false 2023-07-11 14:20:23,873 INFO [Listener at localhost/43015] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-07-11 14:20:23,873 INFO [Listener at localhost/43015] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:23,873 INFO [Listener at localhost/43015] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:23,873 INFO [Listener at localhost/43015] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 14:20:23,874 INFO [Listener at localhost/43015] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:23,874 INFO [Listener at localhost/43015] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 14:20:23,874 INFO [Listener at localhost/43015] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 14:20:23,875 INFO [Listener at localhost/43015] ipc.NettyRpcServer(120): Bind to /172.31.2.10:33199 2023-07-11 14:20:23,875 INFO [Listener at localhost/43015] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-11 14:20:23,876 DEBUG [Listener at localhost/43015] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-11 14:20:23,876 INFO [Listener at localhost/43015] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:23,877 INFO [Listener at localhost/43015] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:23,878 INFO [Listener at localhost/43015] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33199 connecting to ZooKeeper ensemble=127.0.0.1:60113 2023-07-11 14:20:23,881 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): regionserver:331990x0, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 14:20:23,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:33199-0x10154b3e5800002 connected 2023-07-11 14:20:23,882 DEBUG [Listener at localhost/43015] zookeeper.ZKUtil(164): regionserver:33199-0x10154b3e5800002, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:23,882 DEBUG [Listener at localhost/43015] zookeeper.ZKUtil(164): regionserver:33199-0x10154b3e5800002, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:23,883 DEBUG [Listener at localhost/43015] zookeeper.ZKUtil(164): regionserver:33199-0x10154b3e5800002, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 14:20:23,884 DEBUG [Listener at localhost/43015] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33199 2023-07-11 14:20:23,884 DEBUG [Listener at localhost/43015] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33199 2023-07-11 14:20:23,884 DEBUG [Listener at localhost/43015] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33199 2023-07-11 14:20:23,885 DEBUG [Listener at localhost/43015] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33199 2023-07-11 14:20:23,885 DEBUG [Listener at localhost/43015] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33199 2023-07-11 14:20:23,886 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,43201,1689085223814 2023-07-11 14:20:23,886 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,36779,1689085223846 2023-07-11 14:20:23,889 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:23,889 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:23,889 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,43201,1689085223814 2023-07-11 14:20:23,890 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,36779,1689085223846 2023-07-11 14:20:23,891 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:23,891 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:23,891 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): regionserver:33199-0x10154b3e5800002, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:23,891 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:23,891 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:23,891 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:23,893 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,43201,1689085223814 from backup master directory 2023-07-11 14:20:23,893 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:23,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:23,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:23,897 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,43201,1689085223814 2023-07-11 14:20:23,897 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:23,897 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:23,897 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase9.apache.org,43201,1689085223814; waiting to become the next active master 2023-07-11 14:20:23,898 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 14:20:23,898 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase9.apache.org,43201,1689085223814 2023-07-11 14:20:24,336 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/hbase.id with ID: 658a6b35-6574-47c7-9e75-d821301e02c7 2023-07-11 14:20:24,350 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:24,353 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:24,353 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:24,363 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7d4741a7 to 127.0.0.1:60113 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:20:24,367 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@435e62d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:20:24,367 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-11 14:20:24,368 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-11 14:20:24,368 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:20:24,369 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store-tmp 2023-07-11 14:20:24,378 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:24,379 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 14:20:24,379 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:24,379 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:24,379 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 14:20:24,379 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:24,379 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:24,379 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:24,380 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/WALs/jenkins-hbase9.apache.org,43201,1689085223814 2023-07-11 14:20:24,383 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C43201%2C1689085223814, suffix=, logDir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/WALs/jenkins-hbase9.apache.org,43201,1689085223814, archiveDir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/oldWALs, maxLogs=10 2023-07-11 14:20:24,400 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43635,DS-12462c9e-c2e2-4c0d-b8b4-7a67c6de1261,DISK] 2023-07-11 14:20:24,403 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/WALs/jenkins-hbase9.apache.org,43201,1689085223814/jenkins-hbase9.apache.org%2C43201%2C1689085223814.1689085224384 2023-07-11 14:20:24,404 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43635,DS-12462c9e-c2e2-4c0d-b8b4-7a67c6de1261,DISK]] 2023-07-11 14:20:24,404 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:24,404 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:24,404 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:24,404 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:24,406 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:24,408 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-11 14:20:24,409 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-11 14:20:24,409 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:24,410 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:24,411 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:24,414 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:24,416 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:24,417 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65461939, jitterRate=-0.02454109489917755}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 14:20:24,417 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:24,417 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-11 14:20:24,419 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:24,419 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:24,419 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:24,419 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:24,420 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-07-11 14:20:24,420 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-11 14:20:24,420 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-11 14:20:24,422 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-11 14:20:24,432 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-11 14:20:24,433 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-11 14:20:24,433 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-11 14:20:24,433 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-11 14:20:24,433 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-11 14:20:24,435 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:24,435 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:24,436 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-11 14:20:24,436 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-11 14:20:24,437 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-11 14:20:24,439 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:24,439 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): regionserver:33199-0x10154b3e5800002, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:24,439 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:24,440 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:24,439 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:24,440 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase9.apache.org,43201,1689085223814, sessionid=0x10154b3e5800000, setting cluster-up flag (Was=false) 2023-07-11 14:20:24,442 INFO [M:1;jenkins-hbase9:36779] regionserver.HRegionServer(951): ClusterId : 658a6b35-6574-47c7-9e75-d821301e02c7 2023-07-11 14:20:24,443 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:24,443 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:24,448 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-11 14:20:24,449 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,43201,1689085223814 2023-07-11 14:20:24,452 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:24,452 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:24,457 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-11 14:20:24,457 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,43201,1689085223814 2023-07-11 14:20:24,458 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.hbase-snapshot/.tmp 2023-07-11 14:20:24,460 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:24,460 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:24,461 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:24,461 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:24,461 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:24,461 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-07-11 14:20:24,461 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:24,461 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-11 14:20:24,461 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:24,461 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085254461 2023-07-11 14:20:24,461 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-11 14:20:24,461 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-11 14:20:24,462 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-11 14:20:24,462 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-11 14:20:24,462 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-11 14:20:24,462 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-11 14:20:24,462 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:24,462 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-11 14:20:24,463 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-11 14:20:24,463 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-11 14:20:24,463 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-11 14:20:24,463 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-11 14:20:24,463 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:24,463 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085224463,5,FailOnTimeoutGroup] 2023-07-11 14:20:24,463 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-11 14:20:24,463 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085224463,5,FailOnTimeoutGroup] 2023-07-11 14:20:24,463 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:24,463 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-11 14:20:24,463 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:24,463 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:24,464 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-11 14:20:24,487 INFO [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(951): ClusterId : 658a6b35-6574-47c7-9e75-d821301e02c7 2023-07-11 14:20:24,487 DEBUG [RS:0;jenkins-hbase9:33199] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-11 14:20:24,489 DEBUG [RS:0;jenkins-hbase9:33199] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-11 14:20:24,489 DEBUG [RS:0;jenkins-hbase9:33199] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-11 14:20:24,492 DEBUG [RS:0;jenkins-hbase9:33199] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-11 14:20:24,493 DEBUG [RS:0;jenkins-hbase9:33199] zookeeper.ReadOnlyZKClient(139): Connect 0x073c892a to 127.0.0.1:60113 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:20:24,497 DEBUG [RS:0;jenkins-hbase9:33199] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dc4e25a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:20:24,497 DEBUG [RS:0;jenkins-hbase9:33199] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48c66d86, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-11 14:20:24,506 DEBUG [RS:0;jenkins-hbase9:33199] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase9:33199 2023-07-11 14:20:24,506 INFO [RS:0;jenkins-hbase9:33199] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-11 14:20:24,506 INFO [RS:0;jenkins-hbase9:33199] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-11 14:20:24,506 DEBUG [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(1022): About to register with Master. 2023-07-11 14:20:24,507 INFO [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase9.apache.org,43201,1689085223814 with isa=jenkins-hbase9.apache.org/172.31.2.10:33199, startcode=1689085223873 2023-07-11 14:20:24,507 DEBUG [RS:0;jenkins-hbase9:33199] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-11 14:20:24,510 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:54153, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-11 14:20:24,511 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=43201] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,33199,1689085223873 2023-07-11 14:20:24,511 DEBUG [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d 2023-07-11 14:20:24,512 DEBUG [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:44189 2023-07-11 14:20:24,512 DEBUG [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-11 14:20:24,513 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 14:20:24,514 DEBUG [RS:0;jenkins-hbase9:33199] zookeeper.ZKUtil(162): regionserver:33199-0x10154b3e5800002, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33199,1689085223873 2023-07-11 14:20:24,514 WARN [RS:0;jenkins-hbase9:33199] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 14:20:24,514 INFO [RS:0;jenkins-hbase9:33199] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:20:24,514 DEBUG [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(1948): logDir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/WALs/jenkins-hbase9.apache.org,33199,1689085223873 2023-07-11 14:20:24,514 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,33199,1689085223873] 2023-07-11 14:20:24,518 DEBUG [RS:0;jenkins-hbase9:33199] zookeeper.ZKUtil(162): regionserver:33199-0x10154b3e5800002, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33199,1689085223873 2023-07-11 14:20:24,519 DEBUG [RS:0;jenkins-hbase9:33199] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-11 14:20:24,519 INFO [RS:0;jenkins-hbase9:33199] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-11 14:20:24,521 INFO [RS:0;jenkins-hbase9:33199] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-11 14:20:24,521 INFO [RS:0;jenkins-hbase9:33199] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-11 14:20:24,521 INFO [RS:0;jenkins-hbase9:33199] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:24,521 INFO [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-11 14:20:24,523 INFO [RS:0;jenkins-hbase9:33199] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:24,523 DEBUG [RS:0;jenkins-hbase9:33199] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:24,523 DEBUG [RS:0;jenkins-hbase9:33199] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:24,523 DEBUG [RS:0;jenkins-hbase9:33199] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:24,523 DEBUG [RS:0;jenkins-hbase9:33199] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:24,524 DEBUG [RS:0;jenkins-hbase9:33199] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:24,524 DEBUG [RS:0;jenkins-hbase9:33199] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-11 14:20:24,524 DEBUG [RS:0;jenkins-hbase9:33199] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:24,524 DEBUG [RS:0;jenkins-hbase9:33199] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:24,524 DEBUG [RS:0;jenkins-hbase9:33199] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:24,524 DEBUG [RS:0;jenkins-hbase9:33199] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:24,525 INFO [RS:0;jenkins-hbase9:33199] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:24,525 INFO [RS:0;jenkins-hbase9:33199] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:24,525 INFO [RS:0;jenkins-hbase9:33199] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:24,537 INFO [RS:0;jenkins-hbase9:33199] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-11 14:20:24,537 INFO [RS:0;jenkins-hbase9:33199] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,33199,1689085223873-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:24,549 INFO [RS:0;jenkins-hbase9:33199] regionserver.Replication(203): jenkins-hbase9.apache.org,33199,1689085223873 started 2023-07-11 14:20:24,549 INFO [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(1637): Serving as jenkins-hbase9.apache.org,33199,1689085223873, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:33199, sessionid=0x10154b3e5800002 2023-07-11 14:20:24,549 DEBUG [RS:0;jenkins-hbase9:33199] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-11 14:20:24,549 DEBUG [RS:0;jenkins-hbase9:33199] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,33199,1689085223873 2023-07-11 14:20:24,549 DEBUG [RS:0;jenkins-hbase9:33199] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33199,1689085223873' 2023-07-11 14:20:24,549 DEBUG [RS:0;jenkins-hbase9:33199] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-11 14:20:24,550 DEBUG [RS:0;jenkins-hbase9:33199] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-11 14:20:24,550 DEBUG [RS:0;jenkins-hbase9:33199] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-11 14:20:24,550 DEBUG [RS:0;jenkins-hbase9:33199] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-11 14:20:24,550 DEBUG [RS:0;jenkins-hbase9:33199] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,33199,1689085223873 2023-07-11 14:20:24,550 DEBUG [RS:0;jenkins-hbase9:33199] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33199,1689085223873' 2023-07-11 14:20:24,550 DEBUG [RS:0;jenkins-hbase9:33199] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-11 14:20:24,551 DEBUG [RS:0;jenkins-hbase9:33199] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-11 14:20:24,551 DEBUG [RS:0;jenkins-hbase9:33199] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-11 14:20:24,551 INFO [RS:0;jenkins-hbase9:33199] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-11 14:20:24,551 INFO [RS:0;jenkins-hbase9:33199] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-11 14:20:24,654 INFO [RS:0;jenkins-hbase9:33199] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C33199%2C1689085223873, suffix=, logDir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/WALs/jenkins-hbase9.apache.org,33199,1689085223873, archiveDir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/oldWALs, maxLogs=32 2023-07-11 14:20:24,671 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43635,DS-12462c9e-c2e2-4c0d-b8b4-7a67c6de1261,DISK] 2023-07-11 14:20:24,674 INFO [RS:0;jenkins-hbase9:33199] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/WALs/jenkins-hbase9.apache.org,33199,1689085223873/jenkins-hbase9.apache.org%2C33199%2C1689085223873.1689085224655 2023-07-11 14:20:24,674 DEBUG [RS:0;jenkins-hbase9:33199] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43635,DS-12462c9e-c2e2-4c0d-b8b4-7a67c6de1261,DISK]] 2023-07-11 14:20:24,875 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:24,876 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:24,876 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d 2023-07-11 14:20:25,286 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:25,288 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-11 14:20:25,290 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/info 2023-07-11 14:20:25,290 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-11 14:20:25,291 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:25,291 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-11 14:20:25,293 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/rep_barrier 2023-07-11 14:20:25,293 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-11 14:20:25,294 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:25,294 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-11 14:20:25,295 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/table 2023-07-11 14:20:25,296 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-11 14:20:25,296 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:25,297 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740 2023-07-11 14:20:25,298 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740 2023-07-11 14:20:25,301 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-11 14:20:25,303 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-11 14:20:25,305 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:25,306 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60611673, jitterRate=-0.09681569039821625}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-11 14:20:25,306 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-11 14:20:25,306 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-11 14:20:25,306 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-11 14:20:25,306 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-11 14:20:25,306 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-11 14:20:25,306 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-11 14:20:25,306 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-11 14:20:25,306 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-11 14:20:25,308 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:25,308 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-11 14:20:25,308 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-11 14:20:25,310 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-11 14:20:25,311 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-11 14:20:25,461 DEBUG [jenkins-hbase9:43201] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-11 14:20:25,462 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,33199,1689085223873, state=OPENING 2023-07-11 14:20:25,465 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-11 14:20:25,466 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:25,466 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:25,467 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:20:25,467 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,33199,1689085223873}] 2023-07-11 14:20:25,467 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:20:25,620 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,33199,1689085223873 2023-07-11 14:20:25,620 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-11 14:20:25,623 INFO [RS-EventLoopGroup-8-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52710, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-11 14:20:25,628 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-11 14:20:25,628 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:20:25,630 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C33199%2C1689085223873.meta, suffix=.meta, logDir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/WALs/jenkins-hbase9.apache.org,33199,1689085223873, archiveDir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/oldWALs, maxLogs=32 2023-07-11 14:20:25,646 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43635,DS-12462c9e-c2e2-4c0d-b8b4-7a67c6de1261,DISK] 2023-07-11 14:20:25,648 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/WALs/jenkins-hbase9.apache.org,33199,1689085223873/jenkins-hbase9.apache.org%2C33199%2C1689085223873.meta.1689085225631.meta 2023-07-11 14:20:25,648 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43635,DS-12462c9e-c2e2-4c0d-b8b4-7a67c6de1261,DISK]] 2023-07-11 14:20:25,649 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:25,649 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-11 14:20:25,649 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-11 14:20:25,649 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-11 14:20:25,649 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-11 14:20:25,649 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:25,650 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-11 14:20:25,650 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-11 14:20:25,651 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-11 14:20:25,652 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/info 2023-07-11 14:20:25,652 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/info 2023-07-11 14:20:25,653 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-11 14:20:25,653 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:25,653 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-11 14:20:25,654 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/rep_barrier 2023-07-11 14:20:25,654 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/rep_barrier 2023-07-11 14:20:25,655 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-11 14:20:25,655 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:25,656 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-11 14:20:25,657 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/table 2023-07-11 14:20:25,657 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/table 2023-07-11 14:20:25,657 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-11 14:20:25,658 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:25,659 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740 2023-07-11 14:20:25,660 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740 2023-07-11 14:20:25,663 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-11 14:20:25,665 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-11 14:20:25,666 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67545730, jitterRate=0.00650981068611145}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-11 14:20:25,666 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-11 14:20:25,667 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689085225619 2023-07-11 14:20:25,671 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-11 14:20:25,671 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-11 14:20:25,671 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,33199,1689085223873, state=OPEN 2023-07-11 14:20:25,675 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-11 14:20:25,675 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-11 14:20:25,675 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:20:25,675 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:20:25,678 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-11 14:20:25,678 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,33199,1689085223873 in 208 msec 2023-07-11 14:20:25,681 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-11 14:20:25,681 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 370 msec 2023-07-11 14:20:25,683 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.2230 sec 2023-07-11 14:20:25,683 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689085225683, completionTime=-1 2023-07-11 14:20:25,684 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-11 14:20:25,684 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:25,686 DEBUG [hconnection-0x5d740bc3-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 14:20:25,689 INFO [RS-EventLoopGroup-8-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52712, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 14:20:25,691 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:25,691 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085285691 2023-07-11 14:20:25,691 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085345691 2023-07-11 14:20:25,691 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 7 msec 2023-07-11 14:20:25,697 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,43201,1689085223814-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:25,697 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,43201,1689085223814-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:25,697 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,43201,1689085223814-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:25,697 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:43201, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:25,697 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:25,697 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-11 14:20:25,697 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-11 14:20:25,698 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:25,699 DEBUG [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-11 14:20:25,700 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-11 14:20:25,701 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-11 14:20:25,703 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.tmp/data/hbase/namespace/d23bc0bb7e54d3ea41537623f78baf29 2023-07-11 14:20:25,703 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.tmp/data/hbase/namespace/d23bc0bb7e54d3ea41537623f78baf29 empty. 2023-07-11 14:20:25,704 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.tmp/data/hbase/namespace/d23bc0bb7e54d3ea41537623f78baf29 2023-07-11 14:20:25,704 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-11 14:20:26,116 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:26,118 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => d23bc0bb7e54d3ea41537623f78baf29, NAME => 'hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.tmp 2023-07-11 14:20:26,527 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:26,527 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing d23bc0bb7e54d3ea41537623f78baf29, disabling compactions & flushes 2023-07-11 14:20:26,527 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29. 2023-07-11 14:20:26,527 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29. 2023-07-11 14:20:26,527 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29. after waiting 0 ms 2023-07-11 14:20:26,527 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29. 2023-07-11 14:20:26,527 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29. 2023-07-11 14:20:26,527 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for d23bc0bb7e54d3ea41537623f78baf29: 2023-07-11 14:20:26,530 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-11 14:20:26,533 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689085226532"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085226532"}]},"ts":"1689085226532"} 2023-07-11 14:20:26,535 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-11 14:20:26,536 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-11 14:20:26,536 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085226536"}]},"ts":"1689085226536"} 2023-07-11 14:20:26,538 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-11 14:20:26,542 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d23bc0bb7e54d3ea41537623f78baf29, ASSIGN}] 2023-07-11 14:20:26,544 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d23bc0bb7e54d3ea41537623f78baf29, ASSIGN 2023-07-11 14:20:26,545 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=d23bc0bb7e54d3ea41537623f78baf29, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,33199,1689085223873; forceNewPlan=false, retain=false 2023-07-11 14:20:26,696 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=d23bc0bb7e54d3ea41537623f78baf29, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33199,1689085223873 2023-07-11 14:20:26,696 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689085226696"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085226696"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085226696"}]},"ts":"1689085226696"} 2023-07-11 14:20:26,698 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure d23bc0bb7e54d3ea41537623f78baf29, server=jenkins-hbase9.apache.org,33199,1689085223873}] 2023-07-11 14:20:26,855 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29. 2023-07-11 14:20:26,855 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d23bc0bb7e54d3ea41537623f78baf29, NAME => 'hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:26,855 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace d23bc0bb7e54d3ea41537623f78baf29 2023-07-11 14:20:26,856 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:26,856 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for d23bc0bb7e54d3ea41537623f78baf29 2023-07-11 14:20:26,856 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for d23bc0bb7e54d3ea41537623f78baf29 2023-07-11 14:20:26,858 INFO [StoreOpener-d23bc0bb7e54d3ea41537623f78baf29-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region d23bc0bb7e54d3ea41537623f78baf29 2023-07-11 14:20:26,859 DEBUG [StoreOpener-d23bc0bb7e54d3ea41537623f78baf29-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/namespace/d23bc0bb7e54d3ea41537623f78baf29/info 2023-07-11 14:20:26,859 DEBUG [StoreOpener-d23bc0bb7e54d3ea41537623f78baf29-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/namespace/d23bc0bb7e54d3ea41537623f78baf29/info 2023-07-11 14:20:26,860 INFO [StoreOpener-d23bc0bb7e54d3ea41537623f78baf29-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d23bc0bb7e54d3ea41537623f78baf29 columnFamilyName info 2023-07-11 14:20:26,861 INFO [StoreOpener-d23bc0bb7e54d3ea41537623f78baf29-1] regionserver.HStore(310): Store=d23bc0bb7e54d3ea41537623f78baf29/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:26,861 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/namespace/d23bc0bb7e54d3ea41537623f78baf29 2023-07-11 14:20:26,862 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/namespace/d23bc0bb7e54d3ea41537623f78baf29 2023-07-11 14:20:26,865 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for d23bc0bb7e54d3ea41537623f78baf29 2023-07-11 14:20:26,867 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/namespace/d23bc0bb7e54d3ea41537623f78baf29/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:26,868 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened d23bc0bb7e54d3ea41537623f78baf29; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65228173, jitterRate=-0.028024479746818542}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 14:20:26,868 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for d23bc0bb7e54d3ea41537623f78baf29: 2023-07-11 14:20:26,869 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29., pid=6, masterSystemTime=1689085226851 2023-07-11 14:20:26,872 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29. 2023-07-11 14:20:26,872 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29. 2023-07-11 14:20:26,873 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=d23bc0bb7e54d3ea41537623f78baf29, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33199,1689085223873 2023-07-11 14:20:26,873 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689085226872"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085226872"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085226872"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085226872"}]},"ts":"1689085226872"} 2023-07-11 14:20:26,877 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-11 14:20:26,877 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure d23bc0bb7e54d3ea41537623f78baf29, server=jenkins-hbase9.apache.org,33199,1689085223873 in 177 msec 2023-07-11 14:20:26,880 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-11 14:20:26,880 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=d23bc0bb7e54d3ea41537623f78baf29, ASSIGN in 335 msec 2023-07-11 14:20:26,881 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-11 14:20:26,881 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085226881"}]},"ts":"1689085226881"} 2023-07-11 14:20:26,883 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-11 14:20:26,885 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-11 14:20:26,887 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.1880 sec 2023-07-11 14:20:26,900 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-11 14:20:26,902 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-11 14:20:26,902 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:26,902 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:26,906 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:26,918 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-11 14:20:26,920 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-11 14:20:26,923 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 17 msec 2023-07-11 14:20:26,928 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:26,937 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-11 14:20:26,941 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 12 msec 2023-07-11 14:20:26,953 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-11 14:20:26,956 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-11 14:20:26,957 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.059sec 2023-07-11 14:20:26,957 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-11 14:20:26,957 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-11 14:20:26,957 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-11 14:20:26,957 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,43201,1689085223814-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-11 14:20:26,957 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,43201,1689085223814-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-11 14:20:26,959 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-11 14:20:26,989 DEBUG [Listener at localhost/43015] zookeeper.ReadOnlyZKClient(139): Connect 0x6ece80ba to 127.0.0.1:60113 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:20:26,992 DEBUG [Listener at localhost/43015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31e8f416, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:20:26,994 DEBUG [hconnection-0x1bfc8982-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 14:20:26,996 INFO [RS-EventLoopGroup-8-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52718, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 14:20:26,998 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase9.apache.org,43201,1689085223814 2023-07-11 14:20:26,998 WARN [Listener at localhost/43015] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-11 14:20:26,998 WARN [Listener at localhost/43015] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-07-11 14:20:26,999 DEBUG [Listener at localhost/43015] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-11 14:20:27,001 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-11 14:20:27,001 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:27,001 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-11 14:20:27,001 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:27,001 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:27,250 INFO [Listener at localhost/43015] procedure.TestMasterFailoverWithProcedures(306): Restart 0 exec state: pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-11 14:20:27,250 INFO [Listener at localhost/43015] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:20:27,250 INFO [Listener at localhost/43015] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:27,250 INFO [Listener at localhost/43015] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:27,251 ERROR [Listener at localhost/43015] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-6,5,PEWorkerGroup] 2023-07-11 14:20:27,252 INFO [Listener at localhost/43015] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:20:27,252 INFO [Listener at localhost/43015] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:27,252 INFO [Listener at localhost/43015] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:27,252 INFO [Listener at localhost/43015] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:27,253 INFO [Listener at localhost/43015] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:27,254 DEBUG [Listener at localhost/43015] store.ProcedureTree(114): Procedure Procedure(pid=9, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.CreateTableProcedure) stack ids=[] 2023-07-11 14:20:27,254 DEBUG [Listener at localhost/43015] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:27,255 DEBUG [Listener at localhost/43015] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:27,255 DEBUG [Listener at localhost/43015] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:27,255 DEBUG [Listener at localhost/43015] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:27,256 DEBUG [Listener at localhost/43015] procedure2.ProcedureExecutor(468): Loading pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-11 14:20:27,256 DEBUG [Listener at localhost/43015] procedure2.Procedure(966): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCreateWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:27,256 INFO [Listener at localhost/43015] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 3 msec 2023-07-11 14:20:27,257 DEBUG [Listener at localhost/43015] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:27,257 DEBUG [Listener at localhost/43015] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,33199,1689085223873, table=hbase:meta, region=1588230740 2023-07-11 14:20:27,258 INFO [Listener at localhost/43015] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085257258 2023-07-11 14:20:27,258 DEBUG [Listener at localhost/43015] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:27,258 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-11 14:20:27,261 DEBUG [Listener at localhost/43015] assignment.RegionStateStore(147): Load hbase:meta entry region=d23bc0bb7e54d3ea41537623f78baf29, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,33199,1689085223873, regionLocation=jenkins-hbase9.apache.org,33199,1689085223873, openSeqNum=2 2023-07-11 14:20:27,261 INFO [Listener at localhost/43015] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:27,261 INFO [Listener at localhost/43015] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085287261 2023-07-11 14:20:27,261 INFO [Listener at localhost/43015] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085347261 2023-07-11 14:20:27,261 INFO [Listener at localhost/43015] assignment.AssignmentManager(1536): Joined the cluster in 2 msec 2023-07-11 14:20:27,261 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-11 14:20:27,261 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:20:27,263 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-11 14:20:27,266 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.tmp/data/default/testCreateWithFailoverAtStep3/985b7c290ca40bfe9e0571e4969f922a 2023-07-11 14:20:27,266 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.tmp/data/default/testCreateWithFailoverAtStep3/985b7c290ca40bfe9e0571e4969f922a empty. 2023-07-11 14:20:27,267 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.tmp/data/default/testCreateWithFailoverAtStep3/985b7c290ca40bfe9e0571e4969f922a 2023-07-11 14:20:27,267 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCreateWithFailoverAtStep3 regions 2023-07-11 14:20:27,683 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.tmp/data/default/testCreateWithFailoverAtStep3/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:27,684 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(7675): creating {ENCODED => 985b7c290ca40bfe9e0571e4969f922a, NAME => 'testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testCreateWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.tmp 2023-07-11 14:20:28,093 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(866): Instantiated testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:28,093 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1604): Closing 985b7c290ca40bfe9e0571e4969f922a, disabling compactions & flushes 2023-07-11 14:20:28,093 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1626): Closing region testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. 2023-07-11 14:20:28,093 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. 2023-07-11 14:20:28,093 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1714): Acquired close lock on testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. after waiting 0 ms 2023-07-11 14:20:28,093 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1724): Updates disabled for region testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. 2023-07-11 14:20:28,093 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1838): Closed testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. 2023-07-11 14:20:28,093 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1558): Region close journal for 985b7c290ca40bfe9e0571e4969f922a: 2023-07-11 14:20:28,095 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:28,095 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-11 14:20:28,095 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:28,095 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:28,262 INFO [Listener at localhost/43015] procedure.TestMasterFailoverWithProcedures(306): Restart 1 exec state: pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-11 14:20:28,262 INFO [Listener at localhost/43015] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:20:28,262 INFO [Listener at localhost/43015] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:28,262 INFO [Listener at localhost/43015] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:28,262 ERROR [Listener at localhost/43015] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-7,5,PEWorkerGroup] 2023-07-11 14:20:28,262 INFO [Listener at localhost/43015] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:20:28,262 INFO [Listener at localhost/43015] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:28,262 INFO [Listener at localhost/43015] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:28,262 INFO [Listener at localhost/43015] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:28,263 INFO [Listener at localhost/43015] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:28,264 DEBUG [Listener at localhost/43015] store.ProcedureTree(114): Procedure Procedure(pid=9, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.CreateTableProcedure) stack ids=[0] 2023-07-11 14:20:28,264 DEBUG [Listener at localhost/43015] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:28,265 DEBUG [Listener at localhost/43015] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:28,265 DEBUG [Listener at localhost/43015] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:28,265 DEBUG [Listener at localhost/43015] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:28,265 DEBUG [Listener at localhost/43015] procedure2.ProcedureExecutor(468): Loading pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-11 14:20:28,266 DEBUG [Listener at localhost/43015] procedure2.Procedure(966): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testCreateWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:28,266 INFO [Listener at localhost/43015] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-11 14:20:28,266 DEBUG [Listener at localhost/43015] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:28,267 DEBUG [Listener at localhost/43015] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,33199,1689085223873, table=hbase:meta, region=1588230740 2023-07-11 14:20:28,267 INFO [Listener at localhost/43015] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085258267 2023-07-11 14:20:28,268 DEBUG [Listener at localhost/43015] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:28,268 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-11 14:20:28,270 DEBUG [Listener at localhost/43015] assignment.RegionStateStore(147): Load hbase:meta entry region=d23bc0bb7e54d3ea41537623f78baf29, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,33199,1689085223873, regionLocation=jenkins-hbase9.apache.org,33199,1689085223873, openSeqNum=2 2023-07-11 14:20:28,270 INFO [Listener at localhost/43015] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:28,270 INFO [Listener at localhost/43015] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085288270 2023-07-11 14:20:28,271 INFO [Listener at localhost/43015] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085348270 2023-07-11 14:20:28,271 INFO [Listener at localhost/43015] assignment.AssignmentManager(1536): Joined the cluster in 2 msec 2023-07-11 14:20:28,271 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-11 14:20:28,275 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.tmp/data/default/testCreateWithFailoverAtStep3/985b7c290ca40bfe9e0571e4969f922a 2023-07-11 14:20:28,276 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.tmp/data/default/testCreateWithFailoverAtStep3/985b7c290ca40bfe9e0571e4969f922a empty. 2023-07-11 14:20:28,277 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.tmp/data/default/testCreateWithFailoverAtStep3/985b7c290ca40bfe9e0571e4969f922a 2023-07-11 14:20:28,277 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCreateWithFailoverAtStep3 regions 2023-07-11 14:20:28,287 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.tmp/data/default/testCreateWithFailoverAtStep3/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:28,289 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(7675): creating {ENCODED => 985b7c290ca40bfe9e0571e4969f922a, NAME => 'testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testCreateWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.tmp 2023-07-11 14:20:28,697 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(866): Instantiated testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:28,697 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1604): Closing 985b7c290ca40bfe9e0571e4969f922a, disabling compactions & flushes 2023-07-11 14:20:28,697 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1626): Closing region testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. 2023-07-11 14:20:28,697 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. 2023-07-11 14:20:28,697 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1714): Acquired close lock on testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. after waiting 0 ms 2023-07-11 14:20:28,697 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1724): Updates disabled for region testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. 2023-07-11 14:20:28,697 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1838): Closed testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. 2023-07-11 14:20:28,697 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1558): Region close journal for 985b7c290ca40bfe9e0571e4969f922a: 2023-07-11 14:20:28,699 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:20:28,701 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_ADD_TO_META 2023-07-11 14:20:28,703 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689085228702"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085228702"}]},"ts":"1689085228702"} 2023-07-11 14:20:28,704 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-11 14:20:28,704 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:28,705 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-11 14:20:28,705 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:28,705 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:28,771 INFO [Listener at localhost/43015] procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state: pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-11 14:20:28,771 INFO [Listener at localhost/43015] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:20:28,771 INFO [Listener at localhost/43015] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:28,771 INFO [Listener at localhost/43015] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:28,771 ERROR [Listener at localhost/43015] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-07-11 14:20:28,771 INFO [Listener at localhost/43015] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:20:28,771 INFO [Listener at localhost/43015] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:28,771 INFO [Listener at localhost/43015] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:28,772 INFO [Listener at localhost/43015] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:28,772 INFO [Listener at localhost/43015] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:28,773 DEBUG [Listener at localhost/43015] store.ProcedureTree(114): Procedure Procedure(pid=9, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.CreateTableProcedure) stack ids=[0, 1] 2023-07-11 14:20:28,774 DEBUG [Listener at localhost/43015] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:28,774 DEBUG [Listener at localhost/43015] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:28,774 DEBUG [Listener at localhost/43015] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:28,775 DEBUG [Listener at localhost/43015] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:28,775 DEBUG [Listener at localhost/43015] procedure2.ProcedureExecutor(468): Loading pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-11 14:20:28,775 DEBUG [Listener at localhost/43015] procedure2.Procedure(966): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testCreateWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:28,775 INFO [Listener at localhost/43015] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-11 14:20:28,776 DEBUG [Listener at localhost/43015] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:28,777 DEBUG [Listener at localhost/43015] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,33199,1689085223873, table=hbase:meta, region=1588230740 2023-07-11 14:20:28,777 INFO [Listener at localhost/43015] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085258777 2023-07-11 14:20:28,778 DEBUG [Listener at localhost/43015] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:28,778 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-11 14:20:28,781 DEBUG [Listener at localhost/43015] assignment.RegionStateStore(147): Load hbase:meta entry region=d23bc0bb7e54d3ea41537623f78baf29, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,33199,1689085223873, regionLocation=jenkins-hbase9.apache.org,33199,1689085223873, openSeqNum=2 2023-07-11 14:20:28,781 DEBUG [Listener at localhost/43015] assignment.RegionStateStore(147): Load hbase:meta entry region=985b7c290ca40bfe9e0571e4969f922a, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-11 14:20:28,781 INFO [Listener at localhost/43015] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:28,781 INFO [Listener at localhost/43015] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085288781 2023-07-11 14:20:28,781 INFO [Listener at localhost/43015] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085348781 2023-07-11 14:20:28,781 INFO [Listener at localhost/43015] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-11 14:20:28,781 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_ADD_TO_META 2023-07-11 14:20:28,782 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689085228782"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085228782"}]},"ts":"1689085228782"} 2023-07-11 14:20:28,785 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-11 14:20:28,785 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:20:28,787 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-11 14:20:28,787 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCreateWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085228787"}]},"ts":"1689085228787"} 2023-07-11 14:20:28,789 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCreateWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-11 14:20:28,793 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCreateWithFailoverAtStep3, region=985b7c290ca40bfe9e0571e4969f922a, ASSIGN}] 2023-07-11 14:20:28,793 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:28,793 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=9, state=WAITING:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-11 14:20:28,793 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:28,793 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=9, state=WAITING:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:29,031 INFO [Listener at localhost/43015] procedure.TestMasterFailoverWithProcedures(313): Trigger master failover 2023-07-11 14:20:29,032 INFO [Listener at localhost/43015] hbase.MiniHBaseCluster(607): Aborting jenkins-hbase9.apache.org,43201,1689085223814 2023-07-11 14:20:29,032 ERROR [Listener at localhost/43015] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-11 14:20:29,032 ERROR [Listener at localhost/43015] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase9.apache.org,43201,1689085223814: Aborting for tests ***** java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:608) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:411) at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.masterFailover(MasterProcedureTestingUtility.java:135) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testRecoveryAndDoubleExecution(TestMasterFailoverWithProcedures.java:314) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testCreateWithFailoverAtStep(TestMasterFailoverWithProcedures.java:121) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testCreateWithFailover(TestMasterFailoverWithProcedures.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-11 14:20:29,032 INFO [Listener at localhost/43015] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,43201,1689085223814' ***** 2023-07-11 14:20:29,032 INFO [Listener at localhost/43015] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost/43015 2023-07-11 14:20:29,033 DEBUG [M:0;jenkins-hbase9:43201] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72e2dc99, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-11 14:20:29,033 INFO [M:0;jenkins-hbase9:43201] regionserver.HRegionServer(1141): aborting server jenkins-hbase9.apache.org,43201,1689085223814 2023-07-11 14:20:29,033 DEBUG [M:0;jenkins-hbase9:43201] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d4741a7 to 127.0.0.1:60113 2023-07-11 14:20:29,033 DEBUG [M:0;jenkins-hbase9:43201] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:29,034 INFO [M:0;jenkins-hbase9:43201] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,43201,1689085223814; all regions closed. 2023-07-11 14:20:29,034 DEBUG [M:0;jenkins-hbase9:43201] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:29,034 DEBUG [M:0;jenkins-hbase9:43201] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-11 14:20:29,034 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-11 14:20:29,034 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085224463] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085224463,5,FailOnTimeoutGroup] 2023-07-11 14:20:29,034 DEBUG [M:0;jenkins-hbase9:43201] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-11 14:20:29,035 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085224463] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085224463,5,FailOnTimeoutGroup] 2023-07-11 14:20:29,035 INFO [M:0;jenkins-hbase9:43201] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-11 14:20:29,035 INFO [M:0;jenkins-hbase9:43201] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-11 14:20:29,036 INFO [M:0;jenkins-hbase9:43201] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-07-11 14:20:29,036 DEBUG [M:0;jenkins-hbase9:43201] master.HMaster(1512): Stopping service threads 2023-07-11 14:20:29,036 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): regionserver:33199-0x10154b3e5800002, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:29,036 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:29,036 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:29,036 INFO [M:0;jenkins-hbase9:43201] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-11 14:20:29,036 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:29,036 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:29,037 ERROR [M:0;jenkins-hbase9:43201] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:29,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:29,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:29,037 INFO [M:0;jenkins-hbase9:43201] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-11 14:20:29,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33199-0x10154b3e5800002, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:29,037 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-11 14:20:29,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:29,037 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-11 14:20:29,037 DEBUG [M:0;jenkins-hbase9:43201] zookeeper.ZKUtil(398): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-11 14:20:29,037 WARN [M:0;jenkins-hbase9:43201] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-11 14:20:29,037 INFO [M:0;jenkins-hbase9:43201] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:29,038 INFO [M:0;jenkins-hbase9:43201] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-11 14:20:29,040 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:29,041 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:29,041 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): regionserver:33199-0x10154b3e5800002, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:29,041 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:29,041 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:29,042 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:29,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:29,042 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,36779,1689085223846 from backup master directory 2023-07-11 14:20:29,043 DEBUG [M:0;jenkins-hbase9:43201] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 14:20:29,044 INFO [M:0;jenkins-hbase9:43201] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:29,044 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:29,044 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,36779,1689085223846 2023-07-11 14:20:29,044 DEBUG [M:0;jenkins-hbase9:43201] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:29,044 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:29,044 DEBUG [M:0;jenkins-hbase9:43201] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 14:20:29,044 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 14:20:29,044 DEBUG [M:0;jenkins-hbase9:43201] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:29,044 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase9.apache.org,36779,1689085223846 2023-07-11 14:20:29,044 ERROR [M:0;jenkins-hbase9:43201] regionserver.HRegion(1825): Memstore data size is 29049 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:29,045 INFO [M:0;jenkins-hbase9:43201] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:29,045 DEBUG [M:0;jenkins-hbase9:43201] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:29,045 INFO [M:0;jenkins-hbase9:43201] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-11 14:20:29,045 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 14:20:29,045 INFO [M:0;jenkins-hbase9:43201] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:43201 2023-07-11 14:20:29,051 DEBUG [M:0;jenkins-hbase9:43201] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,43201,1689085223814 already deleted, retry=false 2023-07-11 14:20:29,063 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:29,147 WARN [RS-EventLoopGroup-8-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:43201 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:43201 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-11 14:20:29,147 DEBUG [RS-EventLoopGroup-8-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:43201 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:43201 2023-07-11 14:20:29,236 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:29,236 INFO [M:0;jenkins-hbase9:43201] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,43201,1689085223814; zookeeper connection closed. 2023-07-11 14:20:29,236 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:43201-0x10154b3e5800000, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:29,250 DEBUG [RS:0;jenkins-hbase9:33199] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-11 14:20:29,252 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:55713, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-11 14:20:29,253 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36779] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:29,355 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36779] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:29,457 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36779] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:29,485 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x3f0f3ec3 to 127.0.0.1:60113 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:20:29,489 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@711dcff0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:20:29,490 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-11 14:20:29,490 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-11 14:20:29,490 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:20:29,497 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/WALs/jenkins-hbase9.apache.org,43201,1689085223814 to hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/WALs/jenkins-hbase9.apache.org,43201,1689085223814-dead as it is dead 2023-07-11 14:20:29,498 INFO [master/jenkins-hbase9:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/WALs/jenkins-hbase9.apache.org,43201,1689085223814-dead/jenkins-hbase9.apache.org%2C43201%2C1689085223814.1689085224384 2023-07-11 14:20:29,498 INFO [master/jenkins-hbase9:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/WALs/jenkins-hbase9.apache.org,43201,1689085223814-dead/jenkins-hbase9.apache.org%2C43201%2C1689085223814.1689085224384 after 0ms 2023-07-11 14:20:29,499 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/WALs/jenkins-hbase9.apache.org,43201,1689085223814-dead/jenkins-hbase9.apache.org%2C43201%2C1689085223814.1689085224384 to hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C43201%2C1689085223814.1689085224384 2023-07-11 14:20:29,499 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/WALs/jenkins-hbase9.apache.org,43201,1689085223814-dead 2023-07-11 14:20:29,500 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/WALs/jenkins-hbase9.apache.org,36779,1689085223846 2023-07-11 14:20:29,502 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C36779%2C1689085223846, suffix=, logDir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/WALs/jenkins-hbase9.apache.org,36779,1689085223846, archiveDir=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/oldWALs, maxLogs=10 2023-07-11 14:20:29,516 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43635,DS-12462c9e-c2e2-4c0d-b8b4-7a67c6de1261,DISK] 2023-07-11 14:20:29,520 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/WALs/jenkins-hbase9.apache.org,36779,1689085223846/jenkins-hbase9.apache.org%2C36779%2C1689085223846.1689085229502 2023-07-11 14:20:29,521 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43635,DS-12462c9e-c2e2-4c0d-b8b4-7a67c6de1261,DISK]] 2023-07-11 14:20:29,521 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:29,521 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:29,521 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:29,521 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:29,523 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:29,524 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-11 14:20:29,525 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-11 14:20:29,526 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:29,526 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-11 14:20:29,526 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C43201%2C1689085223814.1689085224384 2023-07-11 14:20:29,534 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 76, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=70, path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C43201%2C1689085223814.1689085224384 2023-07-11 14:20:29,534 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=28.37 KB heapSize=34.23 KB 2023-07-11 14:20:29,560 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36779] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:29,662 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36779] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:29,765 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36779] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:29,867 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36779] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:29,946 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=28.37 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d174099daaac4d4786da3501e7fd52f8 2023-07-11 14:20:29,954 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d174099daaac4d4786da3501e7fd52f8 as hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d174099daaac4d4786da3501e7fd52f8 2023-07-11 14:20:29,960 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d174099daaac4d4786da3501e7fd52f8, entries=76, sequenceid=70, filesize=33.2 K 2023-07-11 14:20:29,960 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~28.37 KB/29049, heapSize ~34.21 KB/35032, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 426ms, sequenceid=70, compaction requested=false; wal=null 2023-07-11 14:20:29,961 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C43201%2C1689085223814.1689085224384 2023-07-11 14:20:29,966 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:29,968 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/70.seqid, newMaxSeqId=70, maxSeqId=1 2023-07-11 14:20:29,970 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=71; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66644188, jitterRate=-0.0069242119789123535}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 14:20:29,970 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:29,970 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-11 14:20:29,970 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36779] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:29,971 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:29,971 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:29,971 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:29,972 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:29,975 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] store.ProcedureTree(114): Procedure Procedure(pid=9, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.CreateTableProcedure) stack ids=[0, 1, 2] 2023-07-11 14:20:29,976 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:29,976 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:29,976 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:29,977 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:29,977 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(468): Loading pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-11 14:20:29,977 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.Procedure(966): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testCreateWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:29,977 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-07-11 14:20:29,977 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-11 14:20:29,978 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:29,979 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,33199,1689085223873, table=hbase:meta, region=1588230740 2023-07-11 14:20:29,980 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 1 possibly 'live' servers, and 0 'splitting'. 2023-07-11 14:20:29,981 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,33199,1689085223873 2023-07-11 14:20:29,992 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-11 14:20:29,992 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-11 14:20:29,992 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-11 14:20:29,992 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-11 14:20:29,993 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-11 14:20:29,993 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-11 14:20:29,994 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-11 14:20:29,994 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-11 14:20:29,994 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase9.apache.org,36779,1689085223846, sessionid=0x10154b3e5800001, setting cluster-up flag (Was=true) 2023-07-11 14:20:29,996 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-11 14:20:29,997 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,36779,1689085223846 2023-07-11 14:20:30,001 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-11 14:20:30,002 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,36779,1689085223846 2023-07-11 14:20:30,002 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/.hbase-snapshot/.tmp 2023-07-11 14:20:30,003 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:30,003 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:30,003 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:30,003 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:30,003 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-07-11 14:20:30,003 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:30,003 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-11 14:20:30,003 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:30,004 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085260004 2023-07-11 14:20:30,005 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-11 14:20:30,005 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-11 14:20:30,005 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-11 14:20:30,005 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-11 14:20:30,005 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-11 14:20:30,005 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-11 14:20:30,005 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-11 14:20:30,005 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:30,006 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-11 14:20:30,006 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-11 14:20:30,006 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-11 14:20:30,008 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-11 14:20:30,008 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-11 14:20:30,009 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085230009,5,FailOnTimeoutGroup] 2023-07-11 14:20:30,009 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085230009,5,FailOnTimeoutGroup] 2023-07-11 14:20:30,009 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:30,009 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-11 14:20:30,009 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:30,009 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:30,009 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689085230009, completionTime=-1 2023-07-11 14:20:30,009 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-11 14:20:30,009 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:30,011 DEBUG [hconnection-0x2fc927d3-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 14:20:30,012 INFO [RS-EventLoopGroup-8-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52722, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 14:20:30,015 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=d23bc0bb7e54d3ea41537623f78baf29, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,33199,1689085223873, regionLocation=jenkins-hbase9.apache.org,33199,1689085223873, openSeqNum=2 2023-07-11 14:20:30,015 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=985b7c290ca40bfe9e0571e4969f922a, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-11 14:20:30,015 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:30,015 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085290015 2023-07-11 14:20:30,015 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085350015 2023-07-11 14:20:30,015 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-07-11 14:20:30,025 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,36779,1689085223846-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:30,025 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,36779,1689085223846-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:30,025 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,36779,1689085223846-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:30,025 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:36779, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:30,025 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:30,026 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-11 14:20:30,027 DEBUG [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-11 14:20:30,035 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-11 14:20:30,038 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-11 14:20:30,038 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 0.994sec 2023-07-11 14:20:30,038 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-11 14:20:30,038 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-11 14:20:30,038 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-11 14:20:30,038 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-11 14:20:30,039 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,36779,1689085223846-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-11 14:20:30,039 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,36779,1689085223846-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-11 14:20:30,039 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCreateWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085230038"}]},"ts":"1689085230038"} 2023-07-11 14:20:30,041 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCreateWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-11 14:20:30,041 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-11 14:20:30,044 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCreateWithFailoverAtStep3, region=985b7c290ca40bfe9e0571e4969f922a, ASSIGN}] 2023-07-11 14:20:30,047 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCreateWithFailoverAtStep3, region=985b7c290ca40bfe9e0571e4969f922a, ASSIGN 2023-07-11 14:20:30,048 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCreateWithFailoverAtStep3, region=985b7c290ca40bfe9e0571e4969f922a, ASSIGN; state=CLOSED, location=jenkins-hbase9.apache.org,33199,1689085223873; forceNewPlan=false, retain=false 2023-07-11 14:20:30,199 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=985b7c290ca40bfe9e0571e4969f922a, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33199,1689085223873 2023-07-11 14:20:30,199 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689085230199"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085230199"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085230199"}]},"ts":"1689085230199"} 2023-07-11 14:20:30,202 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 985b7c290ca40bfe9e0571e4969f922a, server=jenkins-hbase9.apache.org,33199,1689085223873}] 2023-07-11 14:20:30,355 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,33199,1689085223873 2023-07-11 14:20:30,355 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-11 14:20:30,357 INFO [RS-EventLoopGroup-8-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52738, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-11 14:20:30,361 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. 2023-07-11 14:20:30,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 985b7c290ca40bfe9e0571e4969f922a, NAME => 'testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:30,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCreateWithFailoverAtStep3 985b7c290ca40bfe9e0571e4969f922a 2023-07-11 14:20:30,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:30,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 985b7c290ca40bfe9e0571e4969f922a 2023-07-11 14:20:30,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 985b7c290ca40bfe9e0571e4969f922a 2023-07-11 14:20:30,363 INFO [StoreOpener-985b7c290ca40bfe9e0571e4969f922a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 985b7c290ca40bfe9e0571e4969f922a 2023-07-11 14:20:30,364 DEBUG [StoreOpener-985b7c290ca40bfe9e0571e4969f922a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/default/testCreateWithFailoverAtStep3/985b7c290ca40bfe9e0571e4969f922a/f1 2023-07-11 14:20:30,364 DEBUG [StoreOpener-985b7c290ca40bfe9e0571e4969f922a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/default/testCreateWithFailoverAtStep3/985b7c290ca40bfe9e0571e4969f922a/f1 2023-07-11 14:20:30,365 INFO [StoreOpener-985b7c290ca40bfe9e0571e4969f922a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 985b7c290ca40bfe9e0571e4969f922a columnFamilyName f1 2023-07-11 14:20:30,365 INFO [StoreOpener-985b7c290ca40bfe9e0571e4969f922a-1] regionserver.HStore(310): Store=985b7c290ca40bfe9e0571e4969f922a/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:30,365 INFO [StoreOpener-985b7c290ca40bfe9e0571e4969f922a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 985b7c290ca40bfe9e0571e4969f922a 2023-07-11 14:20:30,367 DEBUG [StoreOpener-985b7c290ca40bfe9e0571e4969f922a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/default/testCreateWithFailoverAtStep3/985b7c290ca40bfe9e0571e4969f922a/f2 2023-07-11 14:20:30,367 DEBUG [StoreOpener-985b7c290ca40bfe9e0571e4969f922a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/default/testCreateWithFailoverAtStep3/985b7c290ca40bfe9e0571e4969f922a/f2 2023-07-11 14:20:30,367 INFO [StoreOpener-985b7c290ca40bfe9e0571e4969f922a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 985b7c290ca40bfe9e0571e4969f922a columnFamilyName f2 2023-07-11 14:20:30,368 INFO [StoreOpener-985b7c290ca40bfe9e0571e4969f922a-1] regionserver.HStore(310): Store=985b7c290ca40bfe9e0571e4969f922a/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:30,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/default/testCreateWithFailoverAtStep3/985b7c290ca40bfe9e0571e4969f922a 2023-07-11 14:20:30,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/default/testCreateWithFailoverAtStep3/985b7c290ca40bfe9e0571e4969f922a 2023-07-11 14:20:30,371 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testCreateWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:20:30,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 985b7c290ca40bfe9e0571e4969f922a 2023-07-11 14:20:30,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/default/testCreateWithFailoverAtStep3/985b7c290ca40bfe9e0571e4969f922a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:30,375 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 985b7c290ca40bfe9e0571e4969f922a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67980377, jitterRate=0.012986555695533752}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:20:30,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 985b7c290ca40bfe9e0571e4969f922a: 2023-07-11 14:20:30,376 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a., pid=11, masterSystemTime=1689085230355 2023-07-11 14:20:30,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. 2023-07-11 14:20:30,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. 2023-07-11 14:20:30,379 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=985b7c290ca40bfe9e0571e4969f922a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33199,1689085223873 2023-07-11 14:20:30,380 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689085230379"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085230379"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085230379"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085230379"}]},"ts":"1689085230379"} 2023-07-11 14:20:30,383 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-11 14:20:30,383 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 985b7c290ca40bfe9e0571e4969f922a, server=jenkins-hbase9.apache.org,33199,1689085223873 in 179 msec 2023-07-11 14:20:30,385 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-11 14:20:30,385 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testCreateWithFailoverAtStep3, region=985b7c290ca40bfe9e0571e4969f922a, ASSIGN in 339 msec 2023-07-11 14:20:30,386 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-11 14:20:30,386 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCreateWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085230386"}]},"ts":"1689085230386"} 2023-07-11 14:20:30,388 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCreateWithFailoverAtStep3, state=ENABLED in hbase:meta 2023-07-11 14:20:30,390 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_POST_OPERATION 2023-07-11 14:20:30,392 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testCreateWithFailoverAtStep3 in 3.3920 sec 2023-07-11 14:20:30,519 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCreateWithFailoverAtStep3' 2023-07-11 14:20:30,537 DEBUG [Listener at localhost/43015] util.CommonFSUtils(682): File system contents for path hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/default/testCreateWithFailoverAtStep3 2023-07-11 14:20:30,538 DEBUG [Listener at localhost/43015] util.CommonFSUtils(699): |-.tabledesc/ 2023-07-11 14:20:30,539 DEBUG [Listener at localhost/43015] util.CommonFSUtils(702): |----.tableinfo.0000000001 2023-07-11 14:20:30,539 DEBUG [Listener at localhost/43015] util.CommonFSUtils(699): |-.tmp/ 2023-07-11 14:20:30,539 DEBUG [Listener at localhost/43015] util.CommonFSUtils(699): |-985b7c290ca40bfe9e0571e4969f922a/ 2023-07-11 14:20:30,540 DEBUG [Listener at localhost/43015] util.CommonFSUtils(702): |----.regioninfo 2023-07-11 14:20:30,540 DEBUG [Listener at localhost/43015] util.CommonFSUtils(699): |----f1/ 2023-07-11 14:20:30,540 DEBUG [Listener at localhost/43015] util.CommonFSUtils(699): |----f2/ 2023-07-11 14:20:30,541 DEBUG [Listener at localhost/43015] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-11 14:20:30,541 DEBUG [Listener at localhost/43015] util.CommonFSUtils(702): |-------1.seqid 2023-07-11 14:20:30,546 DEBUG [Listener at localhost/43015] procedure.MasterProcedureTestingUtility(212): Table directory layout is as expected. 2023-07-11 14:20:30,551 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-11 14:20:30,551 DEBUG [Listener at localhost/43015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ece80ba to 127.0.0.1:60113 2023-07-11 14:20:30,551 DEBUG [Listener at localhost/43015] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:30,552 DEBUG [Listener at localhost/43015] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-11 14:20:30,552 DEBUG [Listener at localhost/43015] util.JVMClusterUtil(257): Found active master hash=930334582, stopped=false 2023-07-11 14:20:30,552 INFO [Listener at localhost/43015] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase9.apache.org,36779,1689085223846 2023-07-11 14:20:30,554 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:30,554 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): regionserver:33199-0x10154b3e5800002, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:30,554 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:30,554 INFO [Listener at localhost/43015] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:30,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:30,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33199-0x10154b3e5800002, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:30,556 DEBUG [Listener at localhost/43015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f0f3ec3 to 127.0.0.1:60113 2023-07-11 14:20:30,556 DEBUG [Listener at localhost/43015] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:30,556 INFO [Listener at localhost/43015] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,33199,1689085223873' ***** 2023-07-11 14:20:30,556 INFO [Listener at localhost/43015] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-11 14:20:30,556 INFO [RS:0;jenkins-hbase9:33199] regionserver.HeapMemoryManager(220): Stopping 2023-07-11 14:20:30,556 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-11 14:20:30,556 INFO [RS:0;jenkins-hbase9:33199] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-11 14:20:30,557 INFO [RS:0;jenkins-hbase9:33199] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-11 14:20:30,557 INFO [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(3305): Received CLOSE for 985b7c290ca40bfe9e0571e4969f922a 2023-07-11 14:20:30,557 INFO [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(3305): Received CLOSE for d23bc0bb7e54d3ea41537623f78baf29 2023-07-11 14:20:30,557 INFO [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(1144): stopping server jenkins-hbase9.apache.org,33199,1689085223873 2023-07-11 14:20:30,557 DEBUG [RS:0;jenkins-hbase9:33199] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x073c892a to 127.0.0.1:60113 2023-07-11 14:20:30,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 985b7c290ca40bfe9e0571e4969f922a, disabling compactions & flushes 2023-07-11 14:20:30,558 DEBUG [RS:0;jenkins-hbase9:33199] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:30,558 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. 2023-07-11 14:20:30,558 INFO [RS:0;jenkins-hbase9:33199] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-11 14:20:30,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. 2023-07-11 14:20:30,558 INFO [RS:0;jenkins-hbase9:33199] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-11 14:20:30,558 INFO [RS:0;jenkins-hbase9:33199] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-11 14:20:30,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. after waiting 0 ms 2023-07-11 14:20:30,558 INFO [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-11 14:20:30,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. 2023-07-11 14:20:30,558 INFO [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-11 14:20:30,558 DEBUG [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(1478): Online Regions={985b7c290ca40bfe9e0571e4969f922a=testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a., 1588230740=hbase:meta,,1.1588230740, d23bc0bb7e54d3ea41537623f78baf29=hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29.} 2023-07-11 14:20:30,559 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-11 14:20:30,559 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-11 14:20:30,559 DEBUG [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(1504): Waiting on 1588230740, 985b7c290ca40bfe9e0571e4969f922a, d23bc0bb7e54d3ea41537623f78baf29 2023-07-11 14:20:30,559 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-11 14:20:30,559 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-11 14:20:30,559 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-11 14:20:30,559 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=3.09 KB heapSize=5.82 KB 2023-07-11 14:20:30,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/default/testCreateWithFailoverAtStep3/985b7c290ca40bfe9e0571e4969f922a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-11 14:20:30,563 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. 2023-07-11 14:20:30,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 985b7c290ca40bfe9e0571e4969f922a: 2023-07-11 14:20:30,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed testCreateWithFailoverAtStep3,,1689085226998.985b7c290ca40bfe9e0571e4969f922a. 2023-07-11 14:20:30,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing d23bc0bb7e54d3ea41537623f78baf29, disabling compactions & flushes 2023-07-11 14:20:30,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29. 2023-07-11 14:20:30,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29. 2023-07-11 14:20:30,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29. after waiting 0 ms 2023-07-11 14:20:30,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29. 2023-07-11 14:20:30,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing d23bc0bb7e54d3ea41537623f78baf29 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-11 14:20:30,575 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/namespace/d23bc0bb7e54d3ea41537623f78baf29/.tmp/info/8cf030f2e4a84a1c92cc0bb9a063460d 2023-07-11 14:20:30,584 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/namespace/d23bc0bb7e54d3ea41537623f78baf29/.tmp/info/8cf030f2e4a84a1c92cc0bb9a063460d as hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/namespace/d23bc0bb7e54d3ea41537623f78baf29/info/8cf030f2e4a84a1c92cc0bb9a063460d 2023-07-11 14:20:30,591 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/namespace/d23bc0bb7e54d3ea41537623f78baf29/info/8cf030f2e4a84a1c92cc0bb9a063460d, entries=2, sequenceid=6, filesize=4.8 K 2023-07-11 14:20:30,592 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for d23bc0bb7e54d3ea41537623f78baf29 in 28ms, sequenceid=6, compaction requested=false 2023-07-11 14:20:30,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/namespace/d23bc0bb7e54d3ea41537623f78baf29/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-11 14:20:30,597 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29. 2023-07-11 14:20:30,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for d23bc0bb7e54d3ea41537623f78baf29: 2023-07-11 14:20:30,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689085225697.d23bc0bb7e54d3ea41537623f78baf29. 2023-07-11 14:20:30,613 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-11 14:20:30,613 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-11 14:20:30,641 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-11 14:20:30,759 DEBUG [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-11 14:20:30,959 DEBUG [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-11 14:20:30,970 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.82 KB at sequenceid=16 (bloomFilter=false), to=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/.tmp/info/734f6b44fb874c99b8407f888e1d98ba 2023-07-11 14:20:31,156 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-11 14:20:31,159 DEBUG [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-11 14:20:31,360 DEBUG [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-11 14:20:31,386 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=277 B at sequenceid=16 (bloomFilter=false), to=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/.tmp/table/c7ee4c2d72f54cb9a0df57dc7d5566a3 2023-07-11 14:20:31,394 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/.tmp/info/734f6b44fb874c99b8407f888e1d98ba as hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/info/734f6b44fb874c99b8407f888e1d98ba 2023-07-11 14:20:31,400 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/info/734f6b44fb874c99b8407f888e1d98ba, entries=20, sequenceid=16, filesize=7.2 K 2023-07-11 14:20:31,401 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/.tmp/table/c7ee4c2d72f54cb9a0df57dc7d5566a3 as hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/table/c7ee4c2d72f54cb9a0df57dc7d5566a3 2023-07-11 14:20:31,407 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/table/c7ee4c2d72f54cb9a0df57dc7d5566a3, entries=5, sequenceid=16, filesize=4.9 K 2023-07-11 14:20:31,409 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.09 KB/3163, heapSize ~5.54 KB/5672, currentSize=0 B/0 for 1588230740 in 849ms, sequenceid=16, compaction requested=false 2023-07-11 14:20:31,414 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/data/hbase/meta/1588230740/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=1 2023-07-11 14:20:31,415 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-11 14:20:31,415 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-11 14:20:31,415 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-11 14:20:31,416 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-11 14:20:31,560 INFO [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,33199,1689085223873; all regions closed. 2023-07-11 14:20:31,566 DEBUG [RS:0;jenkins-hbase9:33199] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/oldWALs 2023-07-11 14:20:31,566 INFO [RS:0;jenkins-hbase9:33199] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C33199%2C1689085223873.meta:.meta(num 1689085225631) 2023-07-11 14:20:31,570 DEBUG [RS:0;jenkins-hbase9:33199] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/oldWALs 2023-07-11 14:20:31,570 INFO [RS:0;jenkins-hbase9:33199] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C33199%2C1689085223873:(num 1689085224655) 2023-07-11 14:20:31,570 DEBUG [RS:0;jenkins-hbase9:33199] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:31,571 INFO [RS:0;jenkins-hbase9:33199] regionserver.LeaseManager(133): Closed leases 2023-07-11 14:20:31,571 INFO [RS:0;jenkins-hbase9:33199] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-11 14:20:31,571 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 14:20:31,572 INFO [RS:0;jenkins-hbase9:33199] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:33199 2023-07-11 14:20:31,575 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): regionserver:33199-0x10154b3e5800002, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33199,1689085223873 2023-07-11 14:20:31,575 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 14:20:31,575 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): regionserver:33199-0x10154b3e5800002, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 14:20:31,576 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,33199,1689085223873] 2023-07-11 14:20:31,577 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,33199,1689085223873; numProcessing=1 2023-07-11 14:20:31,579 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,33199,1689085223873 already deleted, retry=false 2023-07-11 14:20:31,579 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,33199,1689085223873 expired; onlineServers=0 2023-07-11 14:20:31,579 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,36779,1689085223846' ***** 2023-07-11 14:20:31,579 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-11 14:20:31,579 DEBUG [M:1;jenkins-hbase9:36779] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@334c5540, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-11 14:20:31,579 INFO [M:1;jenkins-hbase9:36779] regionserver.HRegionServer(1144): stopping server jenkins-hbase9.apache.org,36779,1689085223846 2023-07-11 14:20:31,579 INFO [M:1;jenkins-hbase9:36779] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,36779,1689085223846; all regions closed. 2023-07-11 14:20:31,579 DEBUG [M:1;jenkins-hbase9:36779] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:31,579 DEBUG [M:1;jenkins-hbase9:36779] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-11 14:20:31,579 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-11 14:20:31,579 DEBUG [M:1;jenkins-hbase9:36779] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-11 14:20:31,579 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085230009] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085230009,5,FailOnTimeoutGroup] 2023-07-11 14:20:31,579 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085230009] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085230009,5,FailOnTimeoutGroup] 2023-07-11 14:20:31,580 INFO [M:1;jenkins-hbase9:36779] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-11 14:20:31,580 INFO [M:1;jenkins-hbase9:36779] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-11 14:20:31,580 INFO [M:1;jenkins-hbase9:36779] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-07-11 14:20:31,580 DEBUG [M:1;jenkins-hbase9:36779] master.HMaster(1512): Stopping service threads 2023-07-11 14:20:31,580 INFO [M:1;jenkins-hbase9:36779] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-11 14:20:31,581 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:31,581 ERROR [M:1;jenkins-hbase9:36779] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:31,581 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:31,581 INFO [M:1;jenkins-hbase9:36779] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:31,581 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-11 14:20:31,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:31,581 DEBUG [M:1;jenkins-hbase9:36779] zookeeper.ZKUtil(398): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-11 14:20:31,581 WARN [M:1;jenkins-hbase9:36779] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-11 14:20:31,582 INFO [M:1;jenkins-hbase9:36779] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:31,582 INFO [M:1;jenkins-hbase9:36779] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-11 14:20:31,582 DEBUG [M:1;jenkins-hbase9:36779] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 14:20:31,582 INFO [M:1;jenkins-hbase9:36779] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:31,582 DEBUG [M:1;jenkins-hbase9:36779] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:31,582 DEBUG [M:1;jenkins-hbase9:36779] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 14:20:31,582 DEBUG [M:1;jenkins-hbase9:36779] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:31,582 INFO [M:1;jenkins-hbase9:36779] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=11.98 KB heapSize=14.01 KB 2023-07-11 14:20:31,676 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): regionserver:33199-0x10154b3e5800002, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:31,676 INFO [RS:0;jenkins-hbase9:33199] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,33199,1689085223873; zookeeper connection closed. 2023-07-11 14:20:31,676 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): regionserver:33199-0x10154b3e5800002, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:31,677 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@36702d2] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@36702d2 2023-07-11 14:20:31,677 INFO [Listener at localhost/43015] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 1 regionserver(s) complete 2023-07-11 14:20:31,993 INFO [M:1;jenkins-hbase9:36779] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.98 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/3e4af531159f442a8dfbcc8b0e6c9b5a 2023-07-11 14:20:32,000 DEBUG [M:1;jenkins-hbase9:36779] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/3e4af531159f442a8dfbcc8b0e6c9b5a as hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/3e4af531159f442a8dfbcc8b0e6c9b5a 2023-07-11 14:20:32,005 INFO [M:1;jenkins-hbase9:36779] regionserver.HStore(1080): Added hdfs://localhost:44189/user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/3e4af531159f442a8dfbcc8b0e6c9b5a, entries=3, sequenceid=92, filesize=5.7 K 2023-07-11 14:20:32,007 INFO [M:1;jenkins-hbase9:36779] regionserver.HRegion(2948): Finished flush of dataSize ~11.98 KB/12264, heapSize ~13.99 KB/14328, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 425ms, sequenceid=92, compaction requested=false 2023-07-11 14:20:32,008 INFO [M:1;jenkins-hbase9:36779] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:32,008 DEBUG [M:1;jenkins-hbase9:36779] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:32,011 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/7dc0944f-062c-adde-eafa-77c2ea25be1d/MasterData/WALs/jenkins-hbase9.apache.org,36779,1689085223846/jenkins-hbase9.apache.org%2C36779%2C1689085223846.1689085229502 not finished, retry = 0 2023-07-11 14:20:32,112 INFO [M:1;jenkins-hbase9:36779] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-11 14:20:32,112 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 14:20:32,113 INFO [M:1;jenkins-hbase9:36779] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:36779 2023-07-11 14:20:32,115 DEBUG [M:1;jenkins-hbase9:36779] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,36779,1689085223846 already deleted, retry=false 2023-07-11 14:20:32,221 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:32,221 INFO [M:1;jenkins-hbase9:36779] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,36779,1689085223846; zookeeper connection closed. 2023-07-11 14:20:32,221 DEBUG [Listener at localhost/43015-EventThread] zookeeper.ZKWatcher(600): master:36779-0x10154b3e5800001, quorum=127.0.0.1:60113, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:32,222 WARN [Listener at localhost/43015] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-11 14:20:32,225 INFO [Listener at localhost/43015] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-11 14:20:32,329 WARN [BP-919559224-172.31.2.10-1689085222928 heartbeating to localhost/127.0.0.1:44189] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-11 14:20:32,329 WARN [BP-919559224-172.31.2.10-1689085222928 heartbeating to localhost/127.0.0.1:44189] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-919559224-172.31.2.10-1689085222928 (Datanode Uuid c08d77ce-3bb6-49a3-9818-1a102cf00a32) service to localhost/127.0.0.1:44189 2023-07-11 14:20:32,330 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/cluster_e8c9e4a3-15af-961f-cdce-f726bad055a8/dfs/data/data1/current/BP-919559224-172.31.2.10-1689085222928] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-11 14:20:32,330 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/cluster_e8c9e4a3-15af-961f-cdce-f726bad055a8/dfs/data/data2/current/BP-919559224-172.31.2.10-1689085222928] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-11 14:20:32,341 INFO [Listener at localhost/43015] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-11 14:20:32,450 INFO [Listener at localhost/43015] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-11 14:20:32,461 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-07-11 14:20:32,471 INFO [Listener at localhost/43015] hbase.ResourceChecker(175): after: master.procedure.TestMasterFailoverWithProcedures#testCreateWithFailover Thread=84 (was 62) Potentially hanging thread: LeaseRenewer:jenkins@localhost:44189 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/43015 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (200715596) connection to localhost/127.0.0.1:44189 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-5-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'DataNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: IPC Client (200715596) connection to localhost/127.0.0.1:44189 from jenkins.hfs.1 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-7-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.1@localhost:44189 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (200715596) connection to localhost/127.0.0.1:44189 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=417 (was 394) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=298 (was 333), ProcessCount=171 (was 171), AvailableMemoryMB=7645 (was 7749) 2023-07-11 14:20:32,480 INFO [Listener at localhost/43015] hbase.ResourceChecker(147): before: master.procedure.TestMasterFailoverWithProcedures#testDeleteWithFailover Thread=84, OpenFileDescriptor=417, MaxFileDescriptor=60000, SystemLoadAverage=298, ProcessCount=171, AvailableMemoryMB=7644 2023-07-11 14:20:32,480 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=2, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-11 14:20:32,480 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/hadoop.log.dir so I do NOT create it in target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4 2023-07-11 14:20:32,480 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138c8df6-1e5b-bd70-fe06-9867df77ad09/hadoop.tmp.dir so I do NOT create it in target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4 2023-07-11 14:20:32,480 INFO [Listener at localhost/43015] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/cluster_24311b5c-314a-06da-6461-fa1b047acdf1, deleteOnExit=true 2023-07-11 14:20:32,481 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-11 14:20:32,481 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/test.cache.data in system properties and HBase conf 2023-07-11 14:20:32,481 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/hadoop.tmp.dir in system properties and HBase conf 2023-07-11 14:20:32,481 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/hadoop.log.dir in system properties and HBase conf 2023-07-11 14:20:32,481 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-11 14:20:32,482 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-11 14:20:32,482 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-11 14:20:32,482 DEBUG [Listener at localhost/43015] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-11 14:20:32,482 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-11 14:20:32,483 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-11 14:20:32,483 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-11 14:20:32,483 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-11 14:20:32,483 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-11 14:20:32,483 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-11 14:20:32,483 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-11 14:20:32,484 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-11 14:20:32,484 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-11 14:20:32,484 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/nfs.dump.dir in system properties and HBase conf 2023-07-11 14:20:32,484 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/java.io.tmpdir in system properties and HBase conf 2023-07-11 14:20:32,484 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-11 14:20:32,485 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-11 14:20:32,485 INFO [Listener at localhost/43015] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-11 14:20:32,497 WARN [Listener at localhost/43015] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-11 14:20:32,497 WARN [Listener at localhost/43015] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-11 14:20:32,541 WARN [Listener at localhost/43015] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-11 14:20:32,543 INFO [Listener at localhost/43015] log.Slf4jLog(67): jetty-6.1.26 2023-07-11 14:20:32,547 INFO [Listener at localhost/43015] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/java.io.tmpdir/Jetty_localhost_41815_hdfs____bdy3jd/webapp 2023-07-11 14:20:32,640 INFO [Listener at localhost/43015] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41815 2023-07-11 14:20:32,686 WARN [Listener at localhost/43015] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-11 14:20:32,686 WARN [Listener at localhost/43015] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-11 14:20:32,731 WARN [Listener at localhost/39157] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-11 14:20:32,741 WARN [Listener at localhost/39157] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-11 14:20:32,744 WARN [Listener at localhost/39157] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-11 14:20:32,745 INFO [Listener at localhost/39157] log.Slf4jLog(67): jetty-6.1.26 2023-07-11 14:20:32,749 INFO [Listener at localhost/39157] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/java.io.tmpdir/Jetty_localhost_35929_datanode____.tiutmj/webapp 2023-07-11 14:20:32,842 INFO [Listener at localhost/39157] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35929 2023-07-11 14:20:32,849 WARN [Listener at localhost/35859] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-11 14:20:32,950 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc71ba4b71c810786: Processing first storage report for DS-991bdd90-f012-4792-98c9-0baaeddceed0 from datanode d48d5e4a-0a11-4fb2-b4c2-839414cdabf6 2023-07-11 14:20:32,950 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc71ba4b71c810786: from storage DS-991bdd90-f012-4792-98c9-0baaeddceed0 node DatanodeRegistration(127.0.0.1:46215, datanodeUuid=d48d5e4a-0a11-4fb2-b4c2-839414cdabf6, infoPort=45267, infoSecurePort=0, ipcPort=35859, storageInfo=lv=-57;cid=testClusterID;nsid=249566348;c=1689085232504), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-11 14:20:32,950 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc71ba4b71c810786: Processing first storage report for DS-8bfe6e8f-2dd7-43dd-8f5b-c732ae27136e from datanode d48d5e4a-0a11-4fb2-b4c2-839414cdabf6 2023-07-11 14:20:32,950 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc71ba4b71c810786: from storage DS-8bfe6e8f-2dd7-43dd-8f5b-c732ae27136e node DatanodeRegistration(127.0.0.1:46215, datanodeUuid=d48d5e4a-0a11-4fb2-b4c2-839414cdabf6, infoPort=45267, infoSecurePort=0, ipcPort=35859, storageInfo=lv=-57;cid=testClusterID;nsid=249566348;c=1689085232504), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-11 14:20:32,961 DEBUG [Listener at localhost/35859] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4 2023-07-11 14:20:32,964 INFO [Listener at localhost/35859] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/cluster_24311b5c-314a-06da-6461-fa1b047acdf1/zookeeper_0, clientPort=59410, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/cluster_24311b5c-314a-06da-6461-fa1b047acdf1/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/cluster_24311b5c-314a-06da-6461-fa1b047acdf1/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-11 14:20:32,965 INFO [Listener at localhost/35859] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=59410 2023-07-11 14:20:32,965 INFO [Listener at localhost/35859] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:32,966 INFO [Listener at localhost/35859] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:33,377 INFO [Listener at localhost/35859] util.FSUtils(471): Created version file at hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c with version=8 2023-07-11 14:20:33,378 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/hbase-staging 2023-07-11 14:20:33,379 INFO [Listener at localhost/35859] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-07-11 14:20:33,380 INFO [Listener at localhost/35859] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:33,380 INFO [Listener at localhost/35859] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:33,380 INFO [Listener at localhost/35859] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 14:20:33,380 INFO [Listener at localhost/35859] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:33,380 INFO [Listener at localhost/35859] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 14:20:33,380 INFO [Listener at localhost/35859] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 14:20:33,381 INFO [Listener at localhost/35859] ipc.NettyRpcServer(120): Bind to /172.31.2.10:38931 2023-07-11 14:20:33,382 INFO [Listener at localhost/35859] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:33,383 INFO [Listener at localhost/35859] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:33,383 INFO [Listener at localhost/35859] zookeeper.RecoverableZooKeeper(93): Process identifier=master:38931 connecting to ZooKeeper ensemble=127.0.0.1:59410 2023-07-11 14:20:33,391 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:389310x0, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 14:20:33,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:38931-0x10154b40ae30000 connected 2023-07-11 14:20:33,408 DEBUG [Listener at localhost/35859] zookeeper.ZKUtil(164): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:33,409 DEBUG [Listener at localhost/35859] zookeeper.ZKUtil(164): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:33,409 DEBUG [Listener at localhost/35859] zookeeper.ZKUtil(164): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 14:20:33,410 DEBUG [Listener at localhost/35859] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38931 2023-07-11 14:20:33,410 DEBUG [Listener at localhost/35859] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38931 2023-07-11 14:20:33,410 DEBUG [Listener at localhost/35859] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38931 2023-07-11 14:20:33,410 DEBUG [Listener at localhost/35859] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38931 2023-07-11 14:20:33,411 DEBUG [Listener at localhost/35859] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38931 2023-07-11 14:20:33,411 INFO [Listener at localhost/35859] master.HMaster(444): hbase.rootdir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c, hbase.cluster.distributed=false 2023-07-11 14:20:33,412 INFO [Listener at localhost/35859] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-07-11 14:20:33,412 INFO [Listener at localhost/35859] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:33,413 INFO [Listener at localhost/35859] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:33,413 INFO [Listener at localhost/35859] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 14:20:33,413 INFO [Listener at localhost/35859] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:33,413 INFO [Listener at localhost/35859] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 14:20:33,413 INFO [Listener at localhost/35859] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 14:20:33,414 INFO [Listener at localhost/35859] ipc.NettyRpcServer(120): Bind to /172.31.2.10:36159 2023-07-11 14:20:33,415 INFO [Listener at localhost/35859] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:33,416 INFO [Listener at localhost/35859] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:33,416 INFO [Listener at localhost/35859] zookeeper.RecoverableZooKeeper(93): Process identifier=master:36159 connecting to ZooKeeper ensemble=127.0.0.1:59410 2023-07-11 14:20:33,419 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:361590x0, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 14:20:33,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:36159-0x10154b40ae30001 connected 2023-07-11 14:20:33,422 DEBUG [Listener at localhost/35859] zookeeper.ZKUtil(164): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:33,422 DEBUG [Listener at localhost/35859] zookeeper.ZKUtil(164): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:33,423 DEBUG [Listener at localhost/35859] zookeeper.ZKUtil(164): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 14:20:33,429 DEBUG [Listener at localhost/35859] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36159 2023-07-11 14:20:33,429 DEBUG [Listener at localhost/35859] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36159 2023-07-11 14:20:33,429 DEBUG [Listener at localhost/35859] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36159 2023-07-11 14:20:33,430 DEBUG [Listener at localhost/35859] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36159 2023-07-11 14:20:33,430 DEBUG [Listener at localhost/35859] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36159 2023-07-11 14:20:33,430 INFO [Listener at localhost/35859] master.HMaster(444): hbase.rootdir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c, hbase.cluster.distributed=false 2023-07-11 14:20:33,443 INFO [Listener at localhost/35859] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-07-11 14:20:33,443 INFO [Listener at localhost/35859] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:33,443 INFO [Listener at localhost/35859] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:33,443 INFO [Listener at localhost/35859] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 14:20:33,443 INFO [Listener at localhost/35859] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:33,443 INFO [Listener at localhost/35859] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 14:20:33,443 INFO [Listener at localhost/35859] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 14:20:33,444 INFO [Listener at localhost/35859] ipc.NettyRpcServer(120): Bind to /172.31.2.10:37201 2023-07-11 14:20:33,445 INFO [Listener at localhost/35859] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-11 14:20:33,446 DEBUG [Listener at localhost/35859] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-11 14:20:33,446 INFO [Listener at localhost/35859] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:33,447 INFO [Listener at localhost/35859] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:33,448 INFO [Listener at localhost/35859] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37201 connecting to ZooKeeper ensemble=127.0.0.1:59410 2023-07-11 14:20:33,450 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): regionserver:372010x0, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 14:20:33,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:37201-0x10154b40ae30002 connected 2023-07-11 14:20:33,451 DEBUG [Listener at localhost/35859] zookeeper.ZKUtil(164): regionserver:37201-0x10154b40ae30002, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:33,452 DEBUG [Listener at localhost/35859] zookeeper.ZKUtil(164): regionserver:37201-0x10154b40ae30002, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:33,452 DEBUG [Listener at localhost/35859] zookeeper.ZKUtil(164): regionserver:37201-0x10154b40ae30002, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 14:20:33,453 DEBUG [Listener at localhost/35859] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37201 2023-07-11 14:20:33,453 DEBUG [Listener at localhost/35859] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37201 2023-07-11 14:20:33,453 DEBUG [Listener at localhost/35859] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37201 2023-07-11 14:20:33,453 DEBUG [Listener at localhost/35859] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37201 2023-07-11 14:20:33,455 DEBUG [Listener at localhost/35859] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37201 2023-07-11 14:20:33,456 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,38931,1689085233379 2023-07-11 14:20:33,456 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,36159,1689085233412 2023-07-11 14:20:33,459 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:33,459 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:33,459 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,38931,1689085233379 2023-07-11 14:20:33,460 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,36159,1689085233412 2023-07-11 14:20:33,461 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): regionserver:37201-0x10154b40ae30002, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:33,461 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:33,461 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:33,461 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:33,461 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:33,462 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:33,462 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:33,463 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,38931,1689085233379 from backup master directory 2023-07-11 14:20:33,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:33,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:33,463 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase9.apache.org,38931,1689085233379; waiting to become the next active master 2023-07-11 14:20:33,465 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,38931,1689085233379 2023-07-11 14:20:33,465 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:33,465 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:33,465 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 14:20:33,465 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase9.apache.org,38931,1689085233379 2023-07-11 14:20:33,881 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/hbase.id with ID: 62de58bc-91d1-4738-80dc-df004baabf50 2023-07-11 14:20:33,894 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:33,897 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:33,897 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:33,904 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x11aa75c5 to 127.0.0.1:59410 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:20:33,912 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c8ce74f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:20:33,912 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-11 14:20:33,913 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-11 14:20:33,913 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:20:33,914 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store-tmp 2023-07-11 14:20:34,320 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:34,321 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 14:20:34,321 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:34,321 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:34,321 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 14:20:34,321 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:34,321 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:34,321 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:34,322 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/WALs/jenkins-hbase9.apache.org,38931,1689085233379 2023-07-11 14:20:34,324 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C38931%2C1689085233379, suffix=, logDir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/WALs/jenkins-hbase9.apache.org,38931,1689085233379, archiveDir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/oldWALs, maxLogs=10 2023-07-11 14:20:34,340 DEBUG [RS-EventLoopGroup-11-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46215,DS-991bdd90-f012-4792-98c9-0baaeddceed0,DISK] 2023-07-11 14:20:34,342 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/WALs/jenkins-hbase9.apache.org,38931,1689085233379/jenkins-hbase9.apache.org%2C38931%2C1689085233379.1689085234325 2023-07-11 14:20:34,342 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46215,DS-991bdd90-f012-4792-98c9-0baaeddceed0,DISK]] 2023-07-11 14:20:34,342 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:34,343 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:34,343 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:34,343 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:34,345 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:34,347 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-11 14:20:34,348 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-11 14:20:34,348 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:34,349 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:34,349 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:34,353 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:34,358 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:34,359 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67016689, jitterRate=-0.0013735145330429077}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 14:20:34,359 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:34,362 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-11 14:20:34,364 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:34,364 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:34,364 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:34,364 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:34,365 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-07-11 14:20:34,365 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-11 14:20:34,368 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-11 14:20:34,370 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-11 14:20:34,382 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-11 14:20:34,383 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-11 14:20:34,383 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-11 14:20:34,383 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-11 14:20:34,384 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-11 14:20:34,386 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:34,386 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:34,386 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-11 14:20:34,386 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-11 14:20:34,387 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-11 14:20:34,388 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): regionserver:37201-0x10154b40ae30002, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:34,388 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:34,388 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:34,388 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:34,388 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:34,389 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase9.apache.org,38931,1689085233379, sessionid=0x10154b40ae30000, setting cluster-up flag (Was=false) 2023-07-11 14:20:34,392 INFO [M:1;jenkins-hbase9:36159] regionserver.HRegionServer(951): ClusterId : 62de58bc-91d1-4738-80dc-df004baabf50 2023-07-11 14:20:34,393 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:34,393 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:34,400 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-11 14:20:34,401 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,38931,1689085233379 2023-07-11 14:20:34,405 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:34,405 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:34,410 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-11 14:20:34,411 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,38931,1689085233379 2023-07-11 14:20:34,412 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.hbase-snapshot/.tmp 2023-07-11 14:20:34,414 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:34,414 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:34,414 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:34,414 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:34,414 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:34,414 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-07-11 14:20:34,414 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:34,414 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-11 14:20:34,414 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:34,415 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085264415 2023-07-11 14:20:34,415 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-11 14:20:34,415 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-11 14:20:34,415 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-11 14:20:34,416 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-11 14:20:34,416 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-11 14:20:34,416 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-11 14:20:34,416 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:34,416 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-11 14:20:34,416 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:34,416 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-11 14:20:34,417 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-11 14:20:34,417 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-11 14:20:34,417 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-11 14:20:34,417 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-11 14:20:34,417 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085234417,5,FailOnTimeoutGroup] 2023-07-11 14:20:34,417 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085234417,5,FailOnTimeoutGroup] 2023-07-11 14:20:34,417 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:34,417 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-11 14:20:34,417 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:34,417 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:34,418 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-11 14:20:34,457 INFO [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(951): ClusterId : 62de58bc-91d1-4738-80dc-df004baabf50 2023-07-11 14:20:34,458 DEBUG [RS:0;jenkins-hbase9:37201] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-11 14:20:34,459 DEBUG [RS:0;jenkins-hbase9:37201] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-11 14:20:34,459 DEBUG [RS:0;jenkins-hbase9:37201] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-11 14:20:34,461 DEBUG [RS:0;jenkins-hbase9:37201] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-11 14:20:34,463 DEBUG [RS:0;jenkins-hbase9:37201] zookeeper.ReadOnlyZKClient(139): Connect 0x5d92beb6 to 127.0.0.1:59410 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:20:34,467 DEBUG [RS:0;jenkins-hbase9:37201] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@576ad5c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:20:34,467 DEBUG [RS:0;jenkins-hbase9:37201] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1268514f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-11 14:20:34,476 DEBUG [RS:0;jenkins-hbase9:37201] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase9:37201 2023-07-11 14:20:34,476 INFO [RS:0;jenkins-hbase9:37201] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-11 14:20:34,476 INFO [RS:0;jenkins-hbase9:37201] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-11 14:20:34,476 DEBUG [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(1022): About to register with Master. 2023-07-11 14:20:34,477 INFO [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase9.apache.org,38931,1689085233379 with isa=jenkins-hbase9.apache.org/172.31.2.10:37201, startcode=1689085233442 2023-07-11 14:20:34,477 DEBUG [RS:0;jenkins-hbase9:37201] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-11 14:20:34,480 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:59903, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-11 14:20:34,481 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38931] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,37201,1689085233442 2023-07-11 14:20:34,481 DEBUG [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c 2023-07-11 14:20:34,481 DEBUG [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:39157 2023-07-11 14:20:34,481 DEBUG [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-11 14:20:34,488 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 14:20:34,488 DEBUG [RS:0;jenkins-hbase9:37201] zookeeper.ZKUtil(162): regionserver:37201-0x10154b40ae30002, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,37201,1689085233442 2023-07-11 14:20:34,489 WARN [RS:0;jenkins-hbase9:37201] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 14:20:34,489 INFO [RS:0;jenkins-hbase9:37201] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:20:34,489 DEBUG [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(1948): logDir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/WALs/jenkins-hbase9.apache.org,37201,1689085233442 2023-07-11 14:20:34,489 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,37201,1689085233442] 2023-07-11 14:20:34,493 DEBUG [RS:0;jenkins-hbase9:37201] zookeeper.ZKUtil(162): regionserver:37201-0x10154b40ae30002, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,37201,1689085233442 2023-07-11 14:20:34,493 DEBUG [RS:0;jenkins-hbase9:37201] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-11 14:20:34,494 INFO [RS:0;jenkins-hbase9:37201] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-11 14:20:34,498 INFO [RS:0;jenkins-hbase9:37201] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-11 14:20:34,498 INFO [RS:0;jenkins-hbase9:37201] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-11 14:20:34,498 INFO [RS:0;jenkins-hbase9:37201] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:34,498 INFO [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-11 14:20:34,499 INFO [RS:0;jenkins-hbase9:37201] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:34,499 DEBUG [RS:0;jenkins-hbase9:37201] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:34,499 DEBUG [RS:0;jenkins-hbase9:37201] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:34,500 DEBUG [RS:0;jenkins-hbase9:37201] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:34,500 DEBUG [RS:0;jenkins-hbase9:37201] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:34,500 DEBUG [RS:0;jenkins-hbase9:37201] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:34,500 DEBUG [RS:0;jenkins-hbase9:37201] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-11 14:20:34,500 DEBUG [RS:0;jenkins-hbase9:37201] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:34,500 DEBUG [RS:0;jenkins-hbase9:37201] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:34,500 DEBUG [RS:0;jenkins-hbase9:37201] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:34,500 DEBUG [RS:0;jenkins-hbase9:37201] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:34,501 INFO [RS:0;jenkins-hbase9:37201] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:34,501 INFO [RS:0;jenkins-hbase9:37201] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:34,501 INFO [RS:0;jenkins-hbase9:37201] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:34,513 INFO [RS:0;jenkins-hbase9:37201] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-11 14:20:34,513 INFO [RS:0;jenkins-hbase9:37201] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,37201,1689085233442-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:34,525 INFO [RS:0;jenkins-hbase9:37201] regionserver.Replication(203): jenkins-hbase9.apache.org,37201,1689085233442 started 2023-07-11 14:20:34,525 INFO [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(1637): Serving as jenkins-hbase9.apache.org,37201,1689085233442, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:37201, sessionid=0x10154b40ae30002 2023-07-11 14:20:34,525 DEBUG [RS:0;jenkins-hbase9:37201] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-11 14:20:34,526 DEBUG [RS:0;jenkins-hbase9:37201] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,37201,1689085233442 2023-07-11 14:20:34,526 DEBUG [RS:0;jenkins-hbase9:37201] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,37201,1689085233442' 2023-07-11 14:20:34,526 DEBUG [RS:0;jenkins-hbase9:37201] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-11 14:20:34,526 DEBUG [RS:0;jenkins-hbase9:37201] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-11 14:20:34,526 DEBUG [RS:0;jenkins-hbase9:37201] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-11 14:20:34,526 DEBUG [RS:0;jenkins-hbase9:37201] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-11 14:20:34,526 DEBUG [RS:0;jenkins-hbase9:37201] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,37201,1689085233442 2023-07-11 14:20:34,526 DEBUG [RS:0;jenkins-hbase9:37201] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,37201,1689085233442' 2023-07-11 14:20:34,526 DEBUG [RS:0;jenkins-hbase9:37201] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-11 14:20:34,527 DEBUG [RS:0;jenkins-hbase9:37201] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-11 14:20:34,527 DEBUG [RS:0;jenkins-hbase9:37201] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-11 14:20:34,527 INFO [RS:0;jenkins-hbase9:37201] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-11 14:20:34,527 INFO [RS:0;jenkins-hbase9:37201] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-11 14:20:34,629 INFO [RS:0;jenkins-hbase9:37201] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C37201%2C1689085233442, suffix=, logDir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/WALs/jenkins-hbase9.apache.org,37201,1689085233442, archiveDir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/oldWALs, maxLogs=32 2023-07-11 14:20:34,646 DEBUG [RS-EventLoopGroup-11-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46215,DS-991bdd90-f012-4792-98c9-0baaeddceed0,DISK] 2023-07-11 14:20:34,648 INFO [RS:0;jenkins-hbase9:37201] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/WALs/jenkins-hbase9.apache.org,37201,1689085233442/jenkins-hbase9.apache.org%2C37201%2C1689085233442.1689085234630 2023-07-11 14:20:34,648 DEBUG [RS:0;jenkins-hbase9:37201] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46215,DS-991bdd90-f012-4792-98c9-0baaeddceed0,DISK]] 2023-07-11 14:20:34,827 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:34,828 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:34,828 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c 2023-07-11 14:20:35,234 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:35,236 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-11 14:20:35,237 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/info 2023-07-11 14:20:35,238 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-11 14:20:35,238 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:35,238 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-11 14:20:35,239 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/rep_barrier 2023-07-11 14:20:35,240 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-11 14:20:35,240 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:35,240 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-11 14:20:35,242 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/table 2023-07-11 14:20:35,242 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-11 14:20:35,243 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:35,244 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740 2023-07-11 14:20:35,244 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740 2023-07-11 14:20:35,246 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-11 14:20:35,248 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-11 14:20:35,250 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:35,251 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69474477, jitterRate=0.035250380635261536}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-11 14:20:35,251 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-11 14:20:35,251 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-11 14:20:35,251 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-11 14:20:35,251 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-11 14:20:35,251 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-11 14:20:35,251 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-11 14:20:35,251 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-11 14:20:35,251 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-11 14:20:35,253 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:35,253 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-11 14:20:35,253 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-11 14:20:35,254 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-11 14:20:35,255 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-11 14:20:35,405 DEBUG [jenkins-hbase9:38931] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-11 14:20:35,406 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,37201,1689085233442, state=OPENING 2023-07-11 14:20:35,408 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-11 14:20:35,409 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:35,409 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:35,409 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:20:35,409 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:20:35,409 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,37201,1689085233442}] 2023-07-11 14:20:35,562 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,37201,1689085233442 2023-07-11 14:20:35,563 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-11 14:20:35,565 INFO [RS-EventLoopGroup-11-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-11 14:20:35,569 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-11 14:20:35,569 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:20:35,571 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C37201%2C1689085233442.meta, suffix=.meta, logDir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/WALs/jenkins-hbase9.apache.org,37201,1689085233442, archiveDir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/oldWALs, maxLogs=32 2023-07-11 14:20:35,587 DEBUG [RS-EventLoopGroup-11-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46215,DS-991bdd90-f012-4792-98c9-0baaeddceed0,DISK] 2023-07-11 14:20:35,589 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/WALs/jenkins-hbase9.apache.org,37201,1689085233442/jenkins-hbase9.apache.org%2C37201%2C1689085233442.meta.1689085235571.meta 2023-07-11 14:20:35,589 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46215,DS-991bdd90-f012-4792-98c9-0baaeddceed0,DISK]] 2023-07-11 14:20:35,589 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:35,590 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-11 14:20:35,590 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-11 14:20:35,590 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-11 14:20:35,590 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-11 14:20:35,590 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:35,590 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-11 14:20:35,590 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-11 14:20:35,591 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-11 14:20:35,592 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/info 2023-07-11 14:20:35,592 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/info 2023-07-11 14:20:35,593 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-11 14:20:35,593 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:35,593 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-11 14:20:35,594 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/rep_barrier 2023-07-11 14:20:35,594 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/rep_barrier 2023-07-11 14:20:35,595 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-11 14:20:35,595 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:35,595 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-11 14:20:35,596 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/table 2023-07-11 14:20:35,596 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/table 2023-07-11 14:20:35,596 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-11 14:20:35,597 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:35,598 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740 2023-07-11 14:20:35,599 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740 2023-07-11 14:20:35,601 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-11 14:20:35,602 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-11 14:20:35,603 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71515972, jitterRate=0.06567102670669556}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-11 14:20:35,603 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-11 14:20:35,604 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689085235562 2023-07-11 14:20:35,608 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-11 14:20:35,608 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-11 14:20:35,609 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,37201,1689085233442, state=OPEN 2023-07-11 14:20:35,614 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-11 14:20:35,614 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-11 14:20:35,614 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:20:35,614 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:20:35,616 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-11 14:20:35,616 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,37201,1689085233442 in 205 msec 2023-07-11 14:20:35,619 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-11 14:20:35,619 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 363 msec 2023-07-11 14:20:35,621 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.2070 sec 2023-07-11 14:20:35,621 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689085235621, completionTime=-1 2023-07-11 14:20:35,621 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-11 14:20:35,621 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:35,624 DEBUG [hconnection-0x1e33c21e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 14:20:35,626 INFO [RS-EventLoopGroup-11-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38800, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 14:20:35,628 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:35,628 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085295628 2023-07-11 14:20:35,628 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085355628 2023-07-11 14:20:35,628 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-07-11 14:20:35,633 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,38931,1689085233379-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:35,634 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,38931,1689085233379-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:35,634 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,38931,1689085233379-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:35,634 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:38931, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:35,634 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:35,634 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-11 14:20:35,634 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-11 14:20:35,635 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:35,635 DEBUG [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-11 14:20:35,636 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-11 14:20:35,637 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-11 14:20:35,639 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/hbase/namespace/f77b0af169137fecc4e6ad8ded7cc025 2023-07-11 14:20:35,639 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/hbase/namespace/f77b0af169137fecc4e6ad8ded7cc025 empty. 2023-07-11 14:20:35,640 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/hbase/namespace/f77b0af169137fecc4e6ad8ded7cc025 2023-07-11 14:20:35,640 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-11 14:20:36,053 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:36,054 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => f77b0af169137fecc4e6ad8ded7cc025, NAME => 'hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp 2023-07-11 14:20:36,066 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-11 14:20:36,460 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:36,461 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing f77b0af169137fecc4e6ad8ded7cc025, disabling compactions & flushes 2023-07-11 14:20:36,461 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025. 2023-07-11 14:20:36,461 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025. 2023-07-11 14:20:36,461 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025. after waiting 0 ms 2023-07-11 14:20:36,461 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025. 2023-07-11 14:20:36,461 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025. 2023-07-11 14:20:36,461 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for f77b0af169137fecc4e6ad8ded7cc025: 2023-07-11 14:20:36,464 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-11 14:20:36,465 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689085236464"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085236464"}]},"ts":"1689085236464"} 2023-07-11 14:20:36,467 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-11 14:20:36,467 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-11 14:20:36,468 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085236468"}]},"ts":"1689085236468"} 2023-07-11 14:20:36,469 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-11 14:20:36,473 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=f77b0af169137fecc4e6ad8ded7cc025, ASSIGN}] 2023-07-11 14:20:36,475 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=f77b0af169137fecc4e6ad8ded7cc025, ASSIGN 2023-07-11 14:20:36,475 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=f77b0af169137fecc4e6ad8ded7cc025, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,37201,1689085233442; forceNewPlan=false, retain=false 2023-07-11 14:20:36,626 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=f77b0af169137fecc4e6ad8ded7cc025, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,37201,1689085233442 2023-07-11 14:20:36,626 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689085236626"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085236626"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085236626"}]},"ts":"1689085236626"} 2023-07-11 14:20:36,629 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure f77b0af169137fecc4e6ad8ded7cc025, server=jenkins-hbase9.apache.org,37201,1689085233442}] 2023-07-11 14:20:36,784 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025. 2023-07-11 14:20:36,785 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f77b0af169137fecc4e6ad8ded7cc025, NAME => 'hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:36,785 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace f77b0af169137fecc4e6ad8ded7cc025 2023-07-11 14:20:36,785 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:36,785 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for f77b0af169137fecc4e6ad8ded7cc025 2023-07-11 14:20:36,785 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for f77b0af169137fecc4e6ad8ded7cc025 2023-07-11 14:20:36,786 INFO [StoreOpener-f77b0af169137fecc4e6ad8ded7cc025-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region f77b0af169137fecc4e6ad8ded7cc025 2023-07-11 14:20:36,788 DEBUG [StoreOpener-f77b0af169137fecc4e6ad8ded7cc025-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/namespace/f77b0af169137fecc4e6ad8ded7cc025/info 2023-07-11 14:20:36,788 DEBUG [StoreOpener-f77b0af169137fecc4e6ad8ded7cc025-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/namespace/f77b0af169137fecc4e6ad8ded7cc025/info 2023-07-11 14:20:36,788 INFO [StoreOpener-f77b0af169137fecc4e6ad8ded7cc025-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f77b0af169137fecc4e6ad8ded7cc025 columnFamilyName info 2023-07-11 14:20:36,789 INFO [StoreOpener-f77b0af169137fecc4e6ad8ded7cc025-1] regionserver.HStore(310): Store=f77b0af169137fecc4e6ad8ded7cc025/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:36,790 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/namespace/f77b0af169137fecc4e6ad8ded7cc025 2023-07-11 14:20:36,790 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/namespace/f77b0af169137fecc4e6ad8ded7cc025 2023-07-11 14:20:36,792 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for f77b0af169137fecc4e6ad8ded7cc025 2023-07-11 14:20:36,795 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/namespace/f77b0af169137fecc4e6ad8ded7cc025/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:36,795 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened f77b0af169137fecc4e6ad8ded7cc025; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63194447, jitterRate=-0.05832935869693756}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 14:20:36,795 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for f77b0af169137fecc4e6ad8ded7cc025: 2023-07-11 14:20:36,796 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025., pid=6, masterSystemTime=1689085236781 2023-07-11 14:20:36,798 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025. 2023-07-11 14:20:36,798 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025. 2023-07-11 14:20:36,799 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=f77b0af169137fecc4e6ad8ded7cc025, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,37201,1689085233442 2023-07-11 14:20:36,799 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689085236798"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085236798"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085236798"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085236798"}]},"ts":"1689085236798"} 2023-07-11 14:20:36,802 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-11 14:20:36,802 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure f77b0af169137fecc4e6ad8ded7cc025, server=jenkins-hbase9.apache.org,37201,1689085233442 in 172 msec 2023-07-11 14:20:36,804 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-11 14:20:36,804 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=f77b0af169137fecc4e6ad8ded7cc025, ASSIGN in 330 msec 2023-07-11 14:20:36,805 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-11 14:20:36,805 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085236805"}]},"ts":"1689085236805"} 2023-07-11 14:20:36,806 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-11 14:20:36,812 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-11 14:20:36,814 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.1780 sec 2023-07-11 14:20:36,837 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-11 14:20:36,838 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-11 14:20:36,838 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:36,838 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:36,842 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:36,848 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-11 14:20:36,852 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 10 msec 2023-07-11 14:20:36,863 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:36,870 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-11 14:20:36,873 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 9 msec 2023-07-11 14:20:36,878 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-11 14:20:36,880 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-11 14:20:36,880 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.415sec 2023-07-11 14:20:36,880 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-11 14:20:36,880 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-11 14:20:36,880 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-11 14:20:36,880 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,38931,1689085233379-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-11 14:20:36,880 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,38931,1689085233379-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-11 14:20:36,882 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-11 14:20:36,959 DEBUG [Listener at localhost/35859] zookeeper.ReadOnlyZKClient(139): Connect 0x1a6777fe to 127.0.0.1:59410 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:20:36,965 DEBUG [Listener at localhost/35859] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c377065, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:20:36,967 DEBUG [hconnection-0x7bbffbc3-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 14:20:36,968 INFO [RS-EventLoopGroup-11-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38812, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 14:20:36,970 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase9.apache.org,38931,1689085233379 2023-07-11 14:20:36,970 WARN [Listener at localhost/35859] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-11 14:20:36,972 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:36,974 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-11 14:20:36,975 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-11 14:20:36,977 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb 2023-07-11 14:20:36,977 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb empty. 2023-07-11 14:20:36,978 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb 2023-07-11 14:20:36,978 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testDeleteWithFailoverAtStep4 regions 2023-07-11 14:20:37,389 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/default/testDeleteWithFailoverAtStep4/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:37,390 INFO [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(7675): creating {ENCODED => b419278502332d8bee946082253c8deb, NAME => 'testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testDeleteWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp 2023-07-11 14:20:37,797 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(866): Instantiated testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:37,797 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1604): Closing b419278502332d8bee946082253c8deb, disabling compactions & flushes 2023-07-11 14:20:37,797 INFO [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1626): Closing region testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb. 2023-07-11 14:20:37,797 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb. 2023-07-11 14:20:37,797 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1714): Acquired close lock on testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb. after waiting 0 ms 2023-07-11 14:20:37,797 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1724): Updates disabled for region testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb. 2023-07-11 14:20:37,797 INFO [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1838): Closed testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb. 2023-07-11 14:20:37,797 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1558): Region close journal for b419278502332d8bee946082253c8deb: 2023-07-11 14:20:37,799 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_ADD_TO_META 2023-07-11 14:20:37,800 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689085237800"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085237800"}]},"ts":"1689085237800"} 2023-07-11 14:20:37,802 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-11 14:20:37,803 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-11 14:20:37,803 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085237803"}]},"ts":"1689085237803"} 2023-07-11 14:20:37,804 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteWithFailoverAtStep4, state=ENABLING in hbase:meta 2023-07-11 14:20:37,807 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=b419278502332d8bee946082253c8deb, ASSIGN}] 2023-07-11 14:20:37,809 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=b419278502332d8bee946082253c8deb, ASSIGN 2023-07-11 14:20:37,809 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=b419278502332d8bee946082253c8deb, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,37201,1689085233442; forceNewPlan=false, retain=false 2023-07-11 14:20:37,960 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=b419278502332d8bee946082253c8deb, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,37201,1689085233442 2023-07-11 14:20:37,960 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689085237960"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085237960"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085237960"}]},"ts":"1689085237960"} 2023-07-11 14:20:37,963 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure b419278502332d8bee946082253c8deb, server=jenkins-hbase9.apache.org,37201,1689085233442}] 2023-07-11 14:20:38,118 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb. 2023-07-11 14:20:38,118 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b419278502332d8bee946082253c8deb, NAME => 'testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:38,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDeleteWithFailoverAtStep4 b419278502332d8bee946082253c8deb 2023-07-11 14:20:38,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:38,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for b419278502332d8bee946082253c8deb 2023-07-11 14:20:38,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for b419278502332d8bee946082253c8deb 2023-07-11 14:20:38,120 INFO [StoreOpener-b419278502332d8bee946082253c8deb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region b419278502332d8bee946082253c8deb 2023-07-11 14:20:38,121 DEBUG [StoreOpener-b419278502332d8bee946082253c8deb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb/f1 2023-07-11 14:20:38,121 DEBUG [StoreOpener-b419278502332d8bee946082253c8deb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb/f1 2023-07-11 14:20:38,122 INFO [StoreOpener-b419278502332d8bee946082253c8deb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b419278502332d8bee946082253c8deb columnFamilyName f1 2023-07-11 14:20:38,122 INFO [StoreOpener-b419278502332d8bee946082253c8deb-1] regionserver.HStore(310): Store=b419278502332d8bee946082253c8deb/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:38,122 INFO [StoreOpener-b419278502332d8bee946082253c8deb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region b419278502332d8bee946082253c8deb 2023-07-11 14:20:38,124 DEBUG [StoreOpener-b419278502332d8bee946082253c8deb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb/f2 2023-07-11 14:20:38,124 DEBUG [StoreOpener-b419278502332d8bee946082253c8deb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb/f2 2023-07-11 14:20:38,124 INFO [StoreOpener-b419278502332d8bee946082253c8deb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b419278502332d8bee946082253c8deb columnFamilyName f2 2023-07-11 14:20:38,125 INFO [StoreOpener-b419278502332d8bee946082253c8deb-1] regionserver.HStore(310): Store=b419278502332d8bee946082253c8deb/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:38,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb 2023-07-11 14:20:38,126 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb 2023-07-11 14:20:38,128 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testDeleteWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:20:38,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for b419278502332d8bee946082253c8deb 2023-07-11 14:20:38,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:38,132 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened b419278502332d8bee946082253c8deb; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68112563, jitterRate=0.014956280589103699}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:20:38,132 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for b419278502332d8bee946082253c8deb: 2023-07-11 14:20:38,133 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb., pid=11, masterSystemTime=1689085238114 2023-07-11 14:20:38,135 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb. 2023-07-11 14:20:38,135 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb. 2023-07-11 14:20:38,135 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=b419278502332d8bee946082253c8deb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,37201,1689085233442 2023-07-11 14:20:38,136 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689085238135"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085238135"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085238135"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085238135"}]},"ts":"1689085238135"} 2023-07-11 14:20:38,140 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-11 14:20:38,140 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure b419278502332d8bee946082253c8deb, server=jenkins-hbase9.apache.org,37201,1689085233442 in 175 msec 2023-07-11 14:20:38,142 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-11 14:20:38,142 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=b419278502332d8bee946082253c8deb, ASSIGN in 333 msec 2023-07-11 14:20:38,143 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-11 14:20:38,143 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085238143"}]},"ts":"1689085238143"} 2023-07-11 14:20:38,144 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteWithFailoverAtStep4, state=ENABLED in hbase:meta 2023-07-11 14:20:38,146 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_POST_OPERATION 2023-07-11 14:20:38,148 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 in 1.1750 sec 2023-07-11 14:20:38,224 DEBUG [Listener at localhost/35859] util.CommonFSUtils(682): File system contents for path hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/default/testDeleteWithFailoverAtStep4 2023-07-11 14:20:38,224 DEBUG [Listener at localhost/35859] util.CommonFSUtils(699): |-.tabledesc/ 2023-07-11 14:20:38,225 DEBUG [Listener at localhost/35859] util.CommonFSUtils(702): |----.tableinfo.0000000001 2023-07-11 14:20:38,225 DEBUG [Listener at localhost/35859] util.CommonFSUtils(699): |-.tmp/ 2023-07-11 14:20:38,225 DEBUG [Listener at localhost/35859] util.CommonFSUtils(699): |-b419278502332d8bee946082253c8deb/ 2023-07-11 14:20:38,226 DEBUG [Listener at localhost/35859] util.CommonFSUtils(702): |----.regioninfo 2023-07-11 14:20:38,226 DEBUG [Listener at localhost/35859] util.CommonFSUtils(699): |----f1/ 2023-07-11 14:20:38,226 DEBUG [Listener at localhost/35859] util.CommonFSUtils(699): |----f2/ 2023-07-11 14:20:38,227 DEBUG [Listener at localhost/35859] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-11 14:20:38,227 DEBUG [Listener at localhost/35859] util.CommonFSUtils(702): |-------1.seqid 2023-07-11 14:20:38,229 DEBUG [Listener at localhost/35859] procedure.MasterProcedureTestingUtility(212): Table directory layout is as expected. 2023-07-11 14:20:38,242 DEBUG [Listener at localhost/35859] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-11 14:20:38,244 INFO [RS-EventLoopGroup-9-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:40576, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-11 14:20:38,248 INFO [Listener at localhost/35859] client.HBaseAdmin$15(890): Started disable of testDeleteWithFailoverAtStep4 2023-07-11 14:20:38,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38931] master.HMaster$11(2418): Client=jenkins//172.31.2.10 disable testDeleteWithFailoverAtStep4 2023-07-11 14:20:38,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38931] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:38,255 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085238255"}]},"ts":"1689085238255"} 2023-07-11 14:20:38,256 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteWithFailoverAtStep4, state=DISABLING in hbase:meta 2023-07-11 14:20:38,258 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testDeleteWithFailoverAtStep4 to state=DISABLING 2023-07-11 14:20:38,258 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=b419278502332d8bee946082253c8deb, UNASSIGN}] 2023-07-11 14:20:38,259 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=b419278502332d8bee946082253c8deb, UNASSIGN 2023-07-11 14:20:38,260 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=b419278502332d8bee946082253c8deb, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,37201,1689085233442 2023-07-11 14:20:38,260 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689085238260"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085238260"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085238260"}]},"ts":"1689085238260"} 2023-07-11 14:20:38,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38931] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-11 14:20:38,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure b419278502332d8bee946082253c8deb, server=jenkins-hbase9.apache.org,37201,1689085233442}] 2023-07-11 14:20:38,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38931] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-11 14:20:38,413 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(111): Close b419278502332d8bee946082253c8deb 2023-07-11 14:20:38,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing b419278502332d8bee946082253c8deb, disabling compactions & flushes 2023-07-11 14:20:38,414 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb. 2023-07-11 14:20:38,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb. 2023-07-11 14:20:38,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb. after waiting 0 ms 2023-07-11 14:20:38,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb. 2023-07-11 14:20:38,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-11 14:20:38,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb. 2023-07-11 14:20:38,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for b419278502332d8bee946082253c8deb: 2023-07-11 14:20:38,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(149): Closed b419278502332d8bee946082253c8deb 2023-07-11 14:20:38,421 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=b419278502332d8bee946082253c8deb, regionState=CLOSED 2023-07-11 14:20:38,421 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDeleteWithFailoverAtStep4,,1689085236971.b419278502332d8bee946082253c8deb.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689085238421"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085238421"}]},"ts":"1689085238421"} 2023-07-11 14:20:38,424 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-11 14:20:38,424 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure b419278502332d8bee946082253c8deb, server=jenkins-hbase9.apache.org,37201,1689085233442 in 161 msec 2023-07-11 14:20:38,426 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-11 14:20:38,426 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=b419278502332d8bee946082253c8deb, UNASSIGN in 166 msec 2023-07-11 14:20:38,426 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085238426"}]},"ts":"1689085238426"} 2023-07-11 14:20:38,428 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteWithFailoverAtStep4, state=DISABLED in hbase:meta 2023-07-11 14:20:38,430 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testDeleteWithFailoverAtStep4 to state=DISABLED 2023-07-11 14:20:38,431 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 in 179 msec 2023-07-11 14:20:38,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38931] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-11 14:20:38,568 INFO [Listener at localhost/35859] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testDeleteWithFailoverAtStep4, procId: 12 completed 2023-07-11 14:20:38,568 WARN [Listener at localhost/35859] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-07-11 14:20:38,573 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:38,575 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:38,576 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:38,576 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:38,576 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:38,576 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:38,823 INFO [Listener at localhost/35859] procedure.TestMasterFailoverWithProcedures(306): Restart 0 exec state: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:38,824 INFO [Listener at localhost/35859] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:20:38,824 INFO [Listener at localhost/35859] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:38,824 INFO [Listener at localhost/35859] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:38,825 ERROR [Listener at localhost/35859] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:38,825 INFO [Listener at localhost/35859] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:20:38,825 INFO [Listener at localhost/35859] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:38,825 INFO [Listener at localhost/35859] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:38,825 INFO [Listener at localhost/35859] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:38,826 INFO [Listener at localhost/35859] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:38,827 DEBUG [Listener at localhost/35859] store.ProcedureTree(114): Procedure Procedure(pid=15, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure) stack ids=[] 2023-07-11 14:20:38,827 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:38,828 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:38,828 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:38,828 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:38,828 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:38,829 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:38,829 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(468): Loading pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:38,829 DEBUG [Listener at localhost/35859] procedure2.Procedure(966): pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:38,829 INFO [Listener at localhost/35859] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 3 msec 2023-07-11 14:20:38,830 DEBUG [Listener at localhost/35859] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:38,830 DEBUG [Listener at localhost/35859] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,37201,1689085233442, table=hbase:meta, region=1588230740 2023-07-11 14:20:38,833 INFO [Listener at localhost/35859] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085268833 2023-07-11 14:20:38,834 DEBUG [Listener at localhost/35859] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:38,834 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:38,837 DEBUG [Listener at localhost/35859] assignment.RegionStateStore(147): Load hbase:meta entry region=f77b0af169137fecc4e6ad8ded7cc025, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,37201,1689085233442, regionLocation=jenkins-hbase9.apache.org,37201,1689085233442, openSeqNum=2 2023-07-11 14:20:38,837 DEBUG [Listener at localhost/35859] assignment.RegionStateStore(147): Load hbase:meta entry region=b419278502332d8bee946082253c8deb, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,37201,1689085233442, regionLocation=jenkins-hbase9.apache.org,37201,1689085233442, openSeqNum=2 2023-07-11 14:20:38,838 INFO [Listener at localhost/35859] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:38,838 INFO [Listener at localhost/35859] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085298838 2023-07-11 14:20:38,838 INFO [Listener at localhost/35859] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085358838 2023-07-11 14:20:38,838 INFO [Listener at localhost/35859] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-07-11 14:20:38,838 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:38,838 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:20:38,840 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:38,843 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb 2023-07-11 14:20:38,846 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb/f1, FileablePath, hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb/f2, FileablePath, hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb/recovered.edits] 2023-07-11 14:20:38,852 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb/recovered.edits/4.seqid to hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/archive/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb/recovered.edits/4.seqid 2023-07-11 14:20:38,852 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb 2023-07-11 14:20:38,853 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testDeleteWithFailoverAtStep4 regions 2023-07-11 14:20:38,854 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:38,854 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:38,854 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:38,854 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:39,088 INFO [Listener at localhost/35859] procedure.TestMasterFailoverWithProcedures(306): Restart 1 exec state: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,088 INFO [Listener at localhost/35859] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:20:39,088 INFO [Listener at localhost/35859] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:39,088 INFO [Listener at localhost/35859] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:39,088 ERROR [Listener at localhost/35859] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:39,089 INFO [Listener at localhost/35859] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:20:39,089 INFO [Listener at localhost/35859] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:39,089 INFO [Listener at localhost/35859] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:39,089 INFO [Listener at localhost/35859] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:39,090 INFO [Listener at localhost/35859] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:39,091 DEBUG [Listener at localhost/35859] store.ProcedureTree(114): Procedure Procedure(pid=15, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure) stack ids=[0] 2023-07-11 14:20:39,091 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:39,092 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:39,092 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:39,092 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:39,093 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,093 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,093 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(468): Loading pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,093 DEBUG [Listener at localhost/35859] procedure2.Procedure(966): pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:39,094 INFO [Listener at localhost/35859] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 3 msec 2023-07-11 14:20:39,094 DEBUG [Listener at localhost/35859] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:39,095 DEBUG [Listener at localhost/35859] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,37201,1689085233442, table=hbase:meta, region=1588230740 2023-07-11 14:20:39,095 INFO [Listener at localhost/35859] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085269095 2023-07-11 14:20:39,096 DEBUG [Listener at localhost/35859] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:39,096 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,099 DEBUG [Listener at localhost/35859] assignment.RegionStateStore(147): Load hbase:meta entry region=f77b0af169137fecc4e6ad8ded7cc025, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,37201,1689085233442, regionLocation=jenkins-hbase9.apache.org,37201,1689085233442, openSeqNum=2 2023-07-11 14:20:39,099 DEBUG [Listener at localhost/35859] assignment.RegionStateStore(147): Load hbase:meta entry region=b419278502332d8bee946082253c8deb, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,37201,1689085233442, regionLocation=jenkins-hbase9.apache.org,37201,1689085233442, openSeqNum=2 2023-07-11 14:20:39,099 INFO [Listener at localhost/35859] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:39,099 INFO [Listener at localhost/35859] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085299099 2023-07-11 14:20:39,099 INFO [Listener at localhost/35859] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085359099 2023-07-11 14:20:39,099 INFO [Listener at localhost/35859] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-11 14:20:39,099 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,101 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb 2023-07-11 14:20:39,102 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb empty. 2023-07-11 14:20:39,102 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.tmp/data/default/testDeleteWithFailoverAtStep4/b419278502332d8bee946082253c8deb 2023-07-11 14:20:39,102 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testDeleteWithFailoverAtStep4 regions 2023-07-11 14:20:39,104 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:20:39,105 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,126 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testDeleteWithFailoverAtStep4 from hbase:meta 2023-07-11 14:20:39,128 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testDeleteWithFailoverAtStep4' descriptor. 2023-07-11 14:20:39,129 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:39,129 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,129 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:39,129 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:39,349 INFO [Listener at localhost/35859] procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,349 INFO [Listener at localhost/35859] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:20:39,350 INFO [Listener at localhost/35859] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:39,350 INFO [Listener at localhost/35859] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:39,350 ERROR [Listener at localhost/35859] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:39,350 INFO [Listener at localhost/35859] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:20:39,350 INFO [Listener at localhost/35859] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:39,350 INFO [Listener at localhost/35859] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:39,350 INFO [Listener at localhost/35859] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:39,351 INFO [Listener at localhost/35859] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:39,352 DEBUG [Listener at localhost/35859] store.ProcedureTree(114): Procedure Procedure(pid=15, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure) stack ids=[0, 1] 2023-07-11 14:20:39,352 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:39,352 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:39,353 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:39,353 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:39,353 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,353 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,354 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(468): Loading pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,354 DEBUG [Listener at localhost/35859] procedure2.Procedure(966): pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:39,354 INFO [Listener at localhost/35859] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-11 14:20:39,354 DEBUG [Listener at localhost/35859] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:39,355 DEBUG [Listener at localhost/35859] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,37201,1689085233442, table=hbase:meta, region=1588230740 2023-07-11 14:20:39,355 INFO [Listener at localhost/35859] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085269355 2023-07-11 14:20:39,356 DEBUG [Listener at localhost/35859] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:39,356 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,359 DEBUG [Listener at localhost/35859] assignment.RegionStateStore(147): Load hbase:meta entry region=f77b0af169137fecc4e6ad8ded7cc025, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,37201,1689085233442, regionLocation=jenkins-hbase9.apache.org,37201,1689085233442, openSeqNum=2 2023-07-11 14:20:39,359 INFO [Listener at localhost/35859] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:39,359 INFO [Listener at localhost/35859] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085299359 2023-07-11 14:20:39,359 INFO [Listener at localhost/35859] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085359359 2023-07-11 14:20:39,359 INFO [Listener at localhost/35859] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-11 14:20:39,359 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,361 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testDeleteWithFailoverAtStep4' descriptor. 2023-07-11 14:20:39,362 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:20:39,363 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,363 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testDeleteWithFailoverAtStep4' from region states. 2023-07-11 14:20:39,363 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 0 regions from META 2023-07-11 14:20:39,363 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [] 2023-07-11 14:20:39,363 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testDeleteWithFailoverAtStep4' as deleted. 2023-07-11 14:20:39,363 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689085239363"}]},"ts":"9223372036854775807"} 2023-07-11 14:20:39,365 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testDeleteWithFailoverAtStep4 state from META 2023-07-11 14:20:39,367 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:39,368 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,368 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:39,368 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:39,610 INFO [Listener at localhost/35859] procedure.TestMasterFailoverWithProcedures(306): Restart 3 exec state: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,610 INFO [Listener at localhost/35859] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:20:39,610 INFO [Listener at localhost/35859] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:39,610 INFO [Listener at localhost/35859] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:39,610 ERROR [Listener at localhost/35859] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:39,610 INFO [Listener at localhost/35859] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:20:39,610 INFO [Listener at localhost/35859] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:39,610 INFO [Listener at localhost/35859] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:39,610 INFO [Listener at localhost/35859] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:39,611 INFO [Listener at localhost/35859] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:39,612 DEBUG [Listener at localhost/35859] store.ProcedureTree(114): Procedure Procedure(pid=15, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure) stack ids=[0, 1, 2] 2023-07-11 14:20:39,612 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:39,613 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:39,613 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:39,613 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:39,613 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,614 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,614 DEBUG [Listener at localhost/35859] procedure2.ProcedureExecutor(468): Loading pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,614 DEBUG [Listener at localhost/35859] procedure2.Procedure(966): pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:39,614 INFO [Listener at localhost/35859] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 3 msec 2023-07-11 14:20:39,615 DEBUG [Listener at localhost/35859] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:39,615 DEBUG [Listener at localhost/35859] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,37201,1689085233442, table=hbase:meta, region=1588230740 2023-07-11 14:20:39,616 INFO [Listener at localhost/35859] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085269616 2023-07-11 14:20:39,616 DEBUG [Listener at localhost/35859] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:39,617 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,619 DEBUG [Listener at localhost/35859] assignment.RegionStateStore(147): Load hbase:meta entry region=f77b0af169137fecc4e6ad8ded7cc025, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,37201,1689085233442, regionLocation=jenkins-hbase9.apache.org,37201,1689085233442, openSeqNum=2 2023-07-11 14:20:39,619 INFO [Listener at localhost/35859] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:39,619 INFO [Listener at localhost/35859] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085299619 2023-07-11 14:20:39,619 INFO [Listener at localhost/35859] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085359619 2023-07-11 14:20:39,619 INFO [Listener at localhost/35859] assignment.AssignmentManager(1536): Joined the cluster in 2 msec 2023-07-11 14:20:39,619 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,619 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testDeleteWithFailoverAtStep4' from region states. 2023-07-11 14:20:39,619 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 0 regions from META 2023-07-11 14:20:39,619 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [] 2023-07-11 14:20:39,619 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testDeleteWithFailoverAtStep4' as deleted. 2023-07-11 14:20:39,620 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689085239620"}]},"ts":"9223372036854775807"} 2023-07-11 14:20:39,622 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testDeleteWithFailoverAtStep4 state from META 2023-07-11 14:20:39,623 DEBUG [PEWorker-1] zookeeper.RecoverableZooKeeper(172): Node /hbase/table/testDeleteWithFailoverAtStep4 already deleted, retry=false 2023-07-11 14:20:39,623 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:20:39,624 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,624 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:39,625 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=15, state=SUCCESS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:39,625 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:39,625 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=15, state=SUCCESS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:39,869 INFO [Listener at localhost/35859] procedure.TestMasterFailoverWithProcedures(313): Trigger master failover 2023-07-11 14:20:39,870 INFO [Listener at localhost/35859] hbase.MiniHBaseCluster(607): Aborting jenkins-hbase9.apache.org,38931,1689085233379 2023-07-11 14:20:39,870 ERROR [Listener at localhost/35859] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-11 14:20:39,870 ERROR [Listener at localhost/35859] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase9.apache.org,38931,1689085233379: Aborting for tests ***** java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:608) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:411) at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.masterFailover(MasterProcedureTestingUtility.java:135) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testRecoveryAndDoubleExecution(TestMasterFailoverWithProcedures.java:314) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testDeleteWithFailoverAtStep(TestMasterFailoverWithProcedures.java:159) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testDeleteWithFailover(TestMasterFailoverWithProcedures.java:137) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-11 14:20:39,870 INFO [Listener at localhost/35859] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,38931,1689085233379' ***** 2023-07-11 14:20:39,870 INFO [Listener at localhost/35859] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost/35859 2023-07-11 14:20:39,871 DEBUG [M:0;jenkins-hbase9:38931] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6383eea9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-11 14:20:39,871 INFO [M:0;jenkins-hbase9:38931] regionserver.HRegionServer(1141): aborting server jenkins-hbase9.apache.org,38931,1689085233379 2023-07-11 14:20:39,871 DEBUG [M:0;jenkins-hbase9:38931] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x11aa75c5 to 127.0.0.1:59410 2023-07-11 14:20:39,871 DEBUG [M:0;jenkins-hbase9:38931] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:39,871 INFO [M:0;jenkins-hbase9:38931] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,38931,1689085233379; all regions closed. 2023-07-11 14:20:39,871 DEBUG [M:0;jenkins-hbase9:38931] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:39,871 DEBUG [M:0;jenkins-hbase9:38931] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-11 14:20:39,872 DEBUG [M:0;jenkins-hbase9:38931] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-11 14:20:39,872 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085234417] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085234417,5,FailOnTimeoutGroup] 2023-07-11 14:20:39,872 INFO [M:0;jenkins-hbase9:38931] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-11 14:20:39,872 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085234417] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085234417,5,FailOnTimeoutGroup] 2023-07-11 14:20:39,872 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-11 14:20:39,873 INFO [M:0;jenkins-hbase9:38931] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-11 14:20:39,874 INFO [M:0;jenkins-hbase9:38931] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-07-11 14:20:39,874 DEBUG [M:0;jenkins-hbase9:38931] master.HMaster(1512): Stopping service threads 2023-07-11 14:20:39,874 INFO [M:0;jenkins-hbase9:38931] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-11 14:20:39,874 ERROR [M:0;jenkins-hbase9:38931] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:39,874 INFO [M:0;jenkins-hbase9:38931] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-11 14:20:39,874 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-11 14:20:39,875 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:39,875 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): regionserver:37201-0x10154b40ae30002, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:39,875 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:39,875 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:39,875 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:39,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:39,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37201-0x10154b40ae30002, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:39,875 DEBUG [M:0;jenkins-hbase9:38931] zookeeper.ZKUtil(398): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-11 14:20:39,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:39,876 WARN [M:0;jenkins-hbase9:38931] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-11 14:20:39,876 INFO [M:0;jenkins-hbase9:38931] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:39,876 INFO [M:0;jenkins-hbase9:38931] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-11 14:20:39,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:39,876 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-11 14:20:39,878 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:39,878 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): regionserver:37201-0x10154b40ae30002, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:39,878 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:39,878 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:39,878 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:39,878 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:39,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:39,879 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,36159,1689085233412 from backup master directory 2023-07-11 14:20:39,880 DEBUG [M:0;jenkins-hbase9:38931] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 14:20:39,880 INFO [M:0;jenkins-hbase9:38931] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:39,880 DEBUG [M:0;jenkins-hbase9:38931] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:39,880 DEBUG [M:0;jenkins-hbase9:38931] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 14:20:39,880 DEBUG [M:0;jenkins-hbase9:38931] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:39,880 ERROR [M:0;jenkins-hbase9:38931] regionserver.HRegion(1825): Memstore data size is 52357 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:39,880 INFO [M:0;jenkins-hbase9:38931] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:39,880 DEBUG [M:0;jenkins-hbase9:38931] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:39,880 INFO [M:0;jenkins-hbase9:38931] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-11 14:20:39,880 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 14:20:39,880 INFO [M:0;jenkins-hbase9:38931] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:38931 2023-07-11 14:20:39,882 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,36159,1689085233412 2023-07-11 14:20:39,882 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:39,882 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:39,882 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 14:20:39,882 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase9.apache.org,36159,1689085233412 2023-07-11 14:20:39,884 DEBUG [M:0;jenkins-hbase9:38931] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,38931,1689085233379 already deleted, retry=false 2023-07-11 14:20:39,895 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:39,922 WARN [RS-EventLoopGroup-11-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:38931 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:38931 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-11 14:20:39,923 DEBUG [RS-EventLoopGroup-11-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:38931 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:38931 2023-07-11 14:20:40,024 DEBUG [RS:0;jenkins-hbase9:37201] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-11 14:20:40,026 INFO [RS-EventLoopGroup-10-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:43449, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-11 14:20:40,027 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36159] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:40,075 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:40,075 INFO [M:0;jenkins-hbase9:38931] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,38931,1689085233379; zookeeper connection closed. 2023-07-11 14:20:40,075 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:38931-0x10154b40ae30000, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:40,129 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36159] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:40,231 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36159] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:40,309 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x38522eec to 127.0.0.1:59410 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:20:40,316 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2fb4a669, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:20:40,317 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-11 14:20:40,317 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-11 14:20:40,317 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:20:40,323 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/WALs/jenkins-hbase9.apache.org,38931,1689085233379 to hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/WALs/jenkins-hbase9.apache.org,38931,1689085233379-dead as it is dead 2023-07-11 14:20:40,324 INFO [master/jenkins-hbase9:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/WALs/jenkins-hbase9.apache.org,38931,1689085233379-dead/jenkins-hbase9.apache.org%2C38931%2C1689085233379.1689085234325 2023-07-11 14:20:40,324 INFO [master/jenkins-hbase9:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/WALs/jenkins-hbase9.apache.org,38931,1689085233379-dead/jenkins-hbase9.apache.org%2C38931%2C1689085233379.1689085234325 after 0ms 2023-07-11 14:20:40,325 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/WALs/jenkins-hbase9.apache.org,38931,1689085233379-dead/jenkins-hbase9.apache.org%2C38931%2C1689085233379.1689085234325 to hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C38931%2C1689085233379.1689085234325 2023-07-11 14:20:40,325 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/WALs/jenkins-hbase9.apache.org,38931,1689085233379-dead 2023-07-11 14:20:40,325 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/WALs/jenkins-hbase9.apache.org,36159,1689085233412 2023-07-11 14:20:40,327 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C36159%2C1689085233412, suffix=, logDir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/WALs/jenkins-hbase9.apache.org,36159,1689085233412, archiveDir=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/oldWALs, maxLogs=10 2023-07-11 14:20:40,334 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36159] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:40,340 DEBUG [RS-EventLoopGroup-11-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46215,DS-991bdd90-f012-4792-98c9-0baaeddceed0,DISK] 2023-07-11 14:20:40,342 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/WALs/jenkins-hbase9.apache.org,36159,1689085233412/jenkins-hbase9.apache.org%2C36159%2C1689085233412.1689085240327 2023-07-11 14:20:40,342 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46215,DS-991bdd90-f012-4792-98c9-0baaeddceed0,DISK]] 2023-07-11 14:20:40,342 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:40,343 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:40,343 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:40,343 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:40,344 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:40,345 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-11 14:20:40,345 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-11 14:20:40,346 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:40,346 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-11 14:20:40,346 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C38931%2C1689085233379.1689085234325 2023-07-11 14:20:40,352 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 133, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=119, path=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C38931%2C1689085233379.1689085234325 2023-07-11 14:20:40,352 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=51.13 KB heapSize=61.23 KB 2023-07-11 14:20:40,436 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36159] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:40,494 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-11 14:20:40,494 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-11 14:20:40,538 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36159] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:40,640 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36159] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:40,742 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36159] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:40,761 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.13 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/659a90a414f24fa08cdd5bc523da625d 2023-07-11 14:20:40,767 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/659a90a414f24fa08cdd5bc523da625d as hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/659a90a414f24fa08cdd5bc523da625d 2023-07-11 14:20:40,773 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/659a90a414f24fa08cdd5bc523da625d, entries=133, sequenceid=119, filesize=56.0 K 2023-07-11 14:20:40,773 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~51.13 KB/52357, heapSize ~61.21 KB/62680, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 421ms, sequenceid=119, compaction requested=false; wal=null 2023-07-11 14:20:40,774 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C38931%2C1689085233379.1689085234325 2023-07-11 14:20:40,777 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:40,779 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/119.seqid, newMaxSeqId=119, maxSeqId=1 2023-07-11 14:20:40,780 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=120; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67239458, jitterRate=0.0019460022449493408}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 14:20:40,780 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:40,780 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-11 14:20:40,781 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:40,781 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:40,781 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:40,782 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:40,785 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] store.ProcedureTree(114): Procedure Procedure(pid=15, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure) stack ids=[0, 1, 2, 3] 2023-07-11 14:20:40,785 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:40,785 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:40,785 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:40,786 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:40,786 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:40,786 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:40,786 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(468): Loading pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:40,787 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.Procedure(966): pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:40,787 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-07-11 14:20:40,787 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-11 14:20:40,787 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:40,788 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,37201,1689085233442, table=hbase:meta, region=1588230740 2023-07-11 14:20:40,789 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 1 possibly 'live' servers, and 0 'splitting'. 2023-07-11 14:20:40,790 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,37201,1689085233442 2023-07-11 14:20:40,801 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-11 14:20:40,801 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-11 14:20:40,801 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-11 14:20:40,801 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-11 14:20:40,802 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-11 14:20:40,802 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-11 14:20:40,803 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-11 14:20:40,803 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-11 14:20:40,803 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase9.apache.org,36159,1689085233412, sessionid=0x10154b40ae30001, setting cluster-up flag (Was=true) 2023-07-11 14:20:40,805 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-11 14:20:40,806 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,36159,1689085233412 2023-07-11 14:20:40,808 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-11 14:20:40,809 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,36159,1689085233412 2023-07-11 14:20:40,809 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/.hbase-snapshot/.tmp 2023-07-11 14:20:40,810 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:40,810 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:40,810 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:40,810 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:40,810 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-07-11 14:20:40,810 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:40,810 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-11 14:20:40,810 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:40,810 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085270810 2023-07-11 14:20:40,811 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-11 14:20:40,812 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:40,812 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-11 14:20:40,812 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-11 14:20:40,812 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-11 14:20:40,812 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-11 14:20:40,812 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-11 14:20:40,812 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:40,812 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-11 14:20:40,812 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-11 14:20:40,813 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-11 14:20:40,813 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-11 14:20:40,813 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-11 14:20:40,813 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085240813,5,FailOnTimeoutGroup] 2023-07-11 14:20:40,813 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085240813,5,FailOnTimeoutGroup] 2023-07-11 14:20:40,813 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:40,813 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-11 14:20:40,813 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:40,813 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:40,813 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689085240813, completionTime=-1 2023-07-11 14:20:40,813 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-11 14:20:40,813 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:40,815 DEBUG [hconnection-0x53c1b15-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 14:20:40,816 INFO [RS-EventLoopGroup-11-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38824, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 14:20:40,818 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=f77b0af169137fecc4e6ad8ded7cc025, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,37201,1689085233442, regionLocation=jenkins-hbase9.apache.org,37201,1689085233442, openSeqNum=2 2023-07-11 14:20:40,818 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:40,818 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085300818 2023-07-11 14:20:40,818 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085360818 2023-07-11 14:20:40,818 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-07-11 14:20:40,826 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,36159,1689085233412-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:40,826 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,36159,1689085233412-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:40,826 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,36159,1689085233412-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:40,826 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:36159, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:40,826 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:40,827 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-11 14:20:40,828 DEBUG [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-11 14:20:40,835 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-11 14:20:40,839 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-11 14:20:40,839 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 0.957sec 2023-07-11 14:20:40,840 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-11 14:20:40,840 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-11 14:20:40,840 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-11 14:20:40,840 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,36159,1689085233412-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-11 14:20:40,840 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,36159,1689085233412-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-11 14:20:40,842 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-11 14:20:40,890 DEBUG [PEWorker-1] procedure.ProcedureSyncWait(216): waitFor quota manager to be available 2023-07-11 14:20:40,890 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-11 14:20:40,892 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 in 2.3220 sec 2023-07-11 14:20:41,092 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-11 14:20:41,129 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-11 14:20:41,130 INFO [Listener at localhost/35859] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-11 14:20:41,130 DEBUG [Listener at localhost/35859] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a6777fe to 127.0.0.1:59410 2023-07-11 14:20:41,130 DEBUG [Listener at localhost/35859] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:41,130 DEBUG [Listener at localhost/35859] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-11 14:20:41,130 DEBUG [Listener at localhost/35859] util.JVMClusterUtil(257): Found active master hash=437693022, stopped=false 2023-07-11 14:20:41,130 INFO [Listener at localhost/35859] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase9.apache.org,36159,1689085233412 2023-07-11 14:20:41,132 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): regionserver:37201-0x10154b40ae30002, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:41,132 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:41,132 INFO [Listener at localhost/35859] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:41,132 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:41,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37201-0x10154b40ae30002, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:41,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:41,134 DEBUG [Listener at localhost/35859] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38522eec to 127.0.0.1:59410 2023-07-11 14:20:41,134 DEBUG [Listener at localhost/35859] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:41,134 INFO [Listener at localhost/35859] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,37201,1689085233442' ***** 2023-07-11 14:20:41,134 INFO [Listener at localhost/35859] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-11 14:20:41,134 INFO [RS:0;jenkins-hbase9:37201] regionserver.HeapMemoryManager(220): Stopping 2023-07-11 14:20:41,134 INFO [RS:0;jenkins-hbase9:37201] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-11 14:20:41,134 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-11 14:20:41,134 INFO [RS:0;jenkins-hbase9:37201] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-11 14:20:41,135 INFO [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(3305): Received CLOSE for f77b0af169137fecc4e6ad8ded7cc025 2023-07-11 14:20:41,135 INFO [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(1144): stopping server jenkins-hbase9.apache.org,37201,1689085233442 2023-07-11 14:20:41,135 DEBUG [RS:0;jenkins-hbase9:37201] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d92beb6 to 127.0.0.1:59410 2023-07-11 14:20:41,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing f77b0af169137fecc4e6ad8ded7cc025, disabling compactions & flushes 2023-07-11 14:20:41,135 DEBUG [RS:0;jenkins-hbase9:37201] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:41,135 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025. 2023-07-11 14:20:41,135 INFO [RS:0;jenkins-hbase9:37201] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-11 14:20:41,136 INFO [RS:0;jenkins-hbase9:37201] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-11 14:20:41,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025. 2023-07-11 14:20:41,136 INFO [RS:0;jenkins-hbase9:37201] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-11 14:20:41,136 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025. after waiting 0 ms 2023-07-11 14:20:41,136 INFO [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-11 14:20:41,136 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025. 2023-07-11 14:20:41,136 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing f77b0af169137fecc4e6ad8ded7cc025 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-11 14:20:41,136 INFO [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-07-11 14:20:41,136 DEBUG [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(1478): Online Regions={f77b0af169137fecc4e6ad8ded7cc025=hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025., 1588230740=hbase:meta,,1.1588230740} 2023-07-11 14:20:41,136 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-11 14:20:41,136 DEBUG [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(1504): Waiting on 1588230740, f77b0af169137fecc4e6ad8ded7cc025 2023-07-11 14:20:41,137 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-11 14:20:41,137 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-11 14:20:41,137 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-11 14:20:41,137 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-11 14:20:41,137 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=4.00 KB heapSize=7.39 KB 2023-07-11 14:20:41,150 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.35 KB at sequenceid=21 (bloomFilter=false), to=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/.tmp/info/7bee552c17d9420c819ce949d316415b 2023-07-11 14:20:41,155 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7bee552c17d9420c819ce949d316415b 2023-07-11 14:20:41,206 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-11 14:20:41,337 DEBUG [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(1504): Waiting on 1588230740, f77b0af169137fecc4e6ad8ded7cc025 2023-07-11 14:20:41,528 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-11 14:20:41,528 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-11 14:20:41,537 DEBUG [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(1504): Waiting on 1588230740, f77b0af169137fecc4e6ad8ded7cc025 2023-07-11 14:20:41,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/namespace/f77b0af169137fecc4e6ad8ded7cc025/.tmp/info/087873e40149456787e78e8221670e2b 2023-07-11 14:20:41,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/namespace/f77b0af169137fecc4e6ad8ded7cc025/.tmp/info/087873e40149456787e78e8221670e2b as hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/namespace/f77b0af169137fecc4e6ad8ded7cc025/info/087873e40149456787e78e8221670e2b 2023-07-11 14:20:41,562 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=109 B at sequenceid=21 (bloomFilter=false), to=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/.tmp/rep_barrier/a2d3c823465244ce864fa6e0f1886752 2023-07-11 14:20:41,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/namespace/f77b0af169137fecc4e6ad8ded7cc025/info/087873e40149456787e78e8221670e2b, entries=2, sequenceid=6, filesize=4.8 K 2023-07-11 14:20:41,565 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for f77b0af169137fecc4e6ad8ded7cc025 in 429ms, sequenceid=6, compaction requested=false 2023-07-11 14:20:41,569 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a2d3c823465244ce864fa6e0f1886752 2023-07-11 14:20:41,570 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/namespace/f77b0af169137fecc4e6ad8ded7cc025/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-11 14:20:41,571 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025. 2023-07-11 14:20:41,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for f77b0af169137fecc4e6ad8ded7cc025: 2023-07-11 14:20:41,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689085235634.f77b0af169137fecc4e6ad8ded7cc025. 2023-07-11 14:20:41,737 DEBUG [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-11 14:20:41,937 DEBUG [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-11 14:20:41,977 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=559 B at sequenceid=21 (bloomFilter=false), to=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/.tmp/table/b42540129d2f45168759ad28a5e91422 2023-07-11 14:20:41,983 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b42540129d2f45168759ad28a5e91422 2023-07-11 14:20:41,984 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/.tmp/info/7bee552c17d9420c819ce949d316415b as hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/info/7bee552c17d9420c819ce949d316415b 2023-07-11 14:20:41,989 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7bee552c17d9420c819ce949d316415b 2023-07-11 14:20:41,990 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/info/7bee552c17d9420c819ce949d316415b, entries=11, sequenceid=21, filesize=6.2 K 2023-07-11 14:20:41,991 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/.tmp/rep_barrier/a2d3c823465244ce864fa6e0f1886752 as hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/rep_barrier/a2d3c823465244ce864fa6e0f1886752 2023-07-11 14:20:41,996 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a2d3c823465244ce864fa6e0f1886752 2023-07-11 14:20:41,996 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/rep_barrier/a2d3c823465244ce864fa6e0f1886752, entries=1, sequenceid=21, filesize=5.0 K 2023-07-11 14:20:41,997 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/.tmp/table/b42540129d2f45168759ad28a5e91422 as hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/table/b42540129d2f45168759ad28a5e91422 2023-07-11 14:20:42,002 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b42540129d2f45168759ad28a5e91422 2023-07-11 14:20:42,003 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/table/b42540129d2f45168759ad28a5e91422, entries=5, sequenceid=21, filesize=5.1 K 2023-07-11 14:20:42,004 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~4.00 KB/4094, heapSize ~7.34 KB/7520, currentSize=0 B/0 for 1588230740 in 866ms, sequenceid=21, compaction requested=false 2023-07-11 14:20:42,011 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/data/hbase/meta/1588230740/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=1 2023-07-11 14:20:42,011 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-11 14:20:42,012 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-11 14:20:42,012 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-11 14:20:42,012 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-11 14:20:42,138 INFO [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,37201,1689085233442; all regions closed. 2023-07-11 14:20:42,141 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/WALs/jenkins-hbase9.apache.org,37201,1689085233442/jenkins-hbase9.apache.org%2C37201%2C1689085233442.meta.1689085235571.meta not finished, retry = 0 2023-07-11 14:20:42,244 DEBUG [RS:0;jenkins-hbase9:37201] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/oldWALs 2023-07-11 14:20:42,244 INFO [RS:0;jenkins-hbase9:37201] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C37201%2C1689085233442.meta:.meta(num 1689085235571) 2023-07-11 14:20:42,249 DEBUG [RS:0;jenkins-hbase9:37201] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/oldWALs 2023-07-11 14:20:42,249 INFO [RS:0;jenkins-hbase9:37201] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C37201%2C1689085233442:(num 1689085234630) 2023-07-11 14:20:42,249 DEBUG [RS:0;jenkins-hbase9:37201] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:42,249 INFO [RS:0;jenkins-hbase9:37201] regionserver.LeaseManager(133): Closed leases 2023-07-11 14:20:42,249 INFO [RS:0;jenkins-hbase9:37201] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-11 14:20:42,249 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 14:20:42,250 INFO [RS:0;jenkins-hbase9:37201] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:37201 2023-07-11 14:20:42,251 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): regionserver:37201-0x10154b40ae30002, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,37201,1689085233442 2023-07-11 14:20:42,251 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 14:20:42,251 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): regionserver:37201-0x10154b40ae30002, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 14:20:42,253 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,37201,1689085233442] 2023-07-11 14:20:42,253 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,37201,1689085233442; numProcessing=1 2023-07-11 14:20:42,258 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,37201,1689085233442 already deleted, retry=false 2023-07-11 14:20:42,258 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,37201,1689085233442 expired; onlineServers=0 2023-07-11 14:20:42,258 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,36159,1689085233412' ***** 2023-07-11 14:20:42,258 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-11 14:20:42,258 DEBUG [M:1;jenkins-hbase9:36159] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e293dc4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-11 14:20:42,258 INFO [M:1;jenkins-hbase9:36159] regionserver.HRegionServer(1144): stopping server jenkins-hbase9.apache.org,36159,1689085233412 2023-07-11 14:20:42,259 INFO [M:1;jenkins-hbase9:36159] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,36159,1689085233412; all regions closed. 2023-07-11 14:20:42,259 DEBUG [M:1;jenkins-hbase9:36159] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:42,259 DEBUG [M:1;jenkins-hbase9:36159] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-11 14:20:42,259 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-11 14:20:42,259 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085240813] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085240813,5,FailOnTimeoutGroup] 2023-07-11 14:20:42,259 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085240813] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085240813,5,FailOnTimeoutGroup] 2023-07-11 14:20:42,259 DEBUG [M:1;jenkins-hbase9:36159] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-11 14:20:42,260 INFO [M:1;jenkins-hbase9:36159] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-11 14:20:42,260 INFO [M:1;jenkins-hbase9:36159] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-11 14:20:42,260 INFO [M:1;jenkins-hbase9:36159] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-07-11 14:20:42,260 DEBUG [M:1;jenkins-hbase9:36159] master.HMaster(1512): Stopping service threads 2023-07-11 14:20:42,260 INFO [M:1;jenkins-hbase9:36159] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-11 14:20:42,260 ERROR [M:1;jenkins-hbase9:36159] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:42,260 INFO [M:1;jenkins-hbase9:36159] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:42,260 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:42,260 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-11 14:20:42,260 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:42,261 DEBUG [M:1;jenkins-hbase9:36159] zookeeper.ZKUtil(398): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-11 14:20:42,261 WARN [M:1;jenkins-hbase9:36159] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-11 14:20:42,261 INFO [M:1;jenkins-hbase9:36159] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:42,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:42,261 INFO [M:1;jenkins-hbase9:36159] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-11 14:20:42,262 DEBUG [M:1;jenkins-hbase9:36159] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 14:20:42,262 INFO [M:1;jenkins-hbase9:36159] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:42,262 DEBUG [M:1;jenkins-hbase9:36159] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:42,262 DEBUG [M:1;jenkins-hbase9:36159] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 14:20:42,262 DEBUG [M:1;jenkins-hbase9:36159] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:42,262 INFO [M:1;jenkins-hbase9:36159] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=661 B heapSize=1.05 KB 2023-07-11 14:20:42,353 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): regionserver:37201-0x10154b40ae30002, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:42,353 INFO [RS:0;jenkins-hbase9:37201] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,37201,1689085233442; zookeeper connection closed. 2023-07-11 14:20:42,353 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): regionserver:37201-0x10154b40ae30002, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:42,355 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@eab3f72] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@eab3f72 2023-07-11 14:20:42,355 INFO [Listener at localhost/35859] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 1 regionserver(s) complete 2023-07-11 14:20:42,671 INFO [M:1;jenkins-hbase9:36159] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=661 B at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ddeba4c22ec2454687615b7a3a52d3b0 2023-07-11 14:20:42,676 DEBUG [M:1;jenkins-hbase9:36159] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ddeba4c22ec2454687615b7a3a52d3b0 as hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ddeba4c22ec2454687615b7a3a52d3b0 2023-07-11 14:20:42,682 INFO [M:1;jenkins-hbase9:36159] regionserver.HStore(1080): Added hdfs://localhost:39157/user/jenkins/test-data/b53e5785-cf87-601c-aa28-296ee8c6ff6c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ddeba4c22ec2454687615b7a3a52d3b0, entries=1, sequenceid=123, filesize=5.1 K 2023-07-11 14:20:42,683 INFO [M:1;jenkins-hbase9:36159] regionserver.HRegion(2948): Finished flush of dataSize ~661 B/661, heapSize ~1.03 KB/1056, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 421ms, sequenceid=123, compaction requested=false 2023-07-11 14:20:42,684 INFO [M:1;jenkins-hbase9:36159] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:42,684 DEBUG [M:1;jenkins-hbase9:36159] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:42,687 INFO [M:1;jenkins-hbase9:36159] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-11 14:20:42,688 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 14:20:42,688 INFO [M:1;jenkins-hbase9:36159] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:36159 2023-07-11 14:20:42,689 DEBUG [M:1;jenkins-hbase9:36159] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,36159,1689085233412 already deleted, retry=false 2023-07-11 14:20:42,791 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:42,791 INFO [M:1;jenkins-hbase9:36159] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,36159,1689085233412; zookeeper connection closed. 2023-07-11 14:20:42,791 DEBUG [Listener at localhost/35859-EventThread] zookeeper.ZKWatcher(600): master:36159-0x10154b40ae30001, quorum=127.0.0.1:59410, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:42,792 WARN [Listener at localhost/35859] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-11 14:20:42,796 INFO [Listener at localhost/35859] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-11 14:20:42,900 WARN [BP-1906051757-172.31.2.10-1689085232504 heartbeating to localhost/127.0.0.1:39157] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-11 14:20:42,901 WARN [BP-1906051757-172.31.2.10-1689085232504 heartbeating to localhost/127.0.0.1:39157] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1906051757-172.31.2.10-1689085232504 (Datanode Uuid d48d5e4a-0a11-4fb2-b4c2-839414cdabf6) service to localhost/127.0.0.1:39157 2023-07-11 14:20:42,901 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/cluster_24311b5c-314a-06da-6461-fa1b047acdf1/dfs/data/data1/current/BP-1906051757-172.31.2.10-1689085232504] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-11 14:20:42,902 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/cluster_24311b5c-314a-06da-6461-fa1b047acdf1/dfs/data/data2/current/BP-1906051757-172.31.2.10-1689085232504] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-11 14:20:42,912 INFO [Listener at localhost/35859] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-11 14:20:43,021 INFO [Listener at localhost/35859] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-11 14:20:43,030 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-07-11 14:20:43,040 INFO [Listener at localhost/35859] hbase.ResourceChecker(175): after: master.procedure.TestMasterFailoverWithProcedures#testDeleteWithFailover Thread=101 (was 84) Potentially hanging thread: IPC Client (200715596) connection to localhost/127.0.0.1:39157 from jenkins.hfs.2 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-6-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (200715596) connection to localhost/127.0.0.1:39157 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: LeaseRenewer:jenkins@localhost:39157 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (200715596) connection to localhost/127.0.0.1:39157 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-7-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/35859 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:39157 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=446 (was 417) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=260 (was 298), ProcessCount=171 (was 171), AvailableMemoryMB=7472 (was 7644) 2023-07-11 14:20:43,049 INFO [Listener at localhost/35859] hbase.ResourceChecker(147): before: master.procedure.TestMasterFailoverWithProcedures#testTruncateWithFailover Thread=101, OpenFileDescriptor=446, MaxFileDescriptor=60000, SystemLoadAverage=260, ProcessCount=171, AvailableMemoryMB=7472 2023-07-11 14:20:43,049 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=2, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-11 14:20:43,049 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/hadoop.log.dir so I do NOT create it in target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa 2023-07-11 14:20:43,049 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b083f83e-8e4b-ce46-1258-afcb24181ce4/hadoop.tmp.dir so I do NOT create it in target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa 2023-07-11 14:20:43,049 INFO [Listener at localhost/35859] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/cluster_ef34d284-81d2-8651-5312-c5ca43158945, deleteOnExit=true 2023-07-11 14:20:43,049 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-11 14:20:43,050 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/test.cache.data in system properties and HBase conf 2023-07-11 14:20:43,050 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/hadoop.tmp.dir in system properties and HBase conf 2023-07-11 14:20:43,050 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/hadoop.log.dir in system properties and HBase conf 2023-07-11 14:20:43,050 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-11 14:20:43,050 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-11 14:20:43,050 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-11 14:20:43,051 DEBUG [Listener at localhost/35859] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-11 14:20:43,051 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-11 14:20:43,051 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-11 14:20:43,051 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-11 14:20:43,051 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-11 14:20:43,052 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-11 14:20:43,052 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-11 14:20:43,052 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-11 14:20:43,052 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-11 14:20:43,052 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-11 14:20:43,052 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/nfs.dump.dir in system properties and HBase conf 2023-07-11 14:20:43,052 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/java.io.tmpdir in system properties and HBase conf 2023-07-11 14:20:43,053 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-11 14:20:43,053 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-11 14:20:43,053 INFO [Listener at localhost/35859] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-11 14:20:43,058 WARN [Listener at localhost/35859] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-11 14:20:43,058 WARN [Listener at localhost/35859] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-11 14:20:43,099 WARN [Listener at localhost/35859] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-11 14:20:43,101 INFO [Listener at localhost/35859] log.Slf4jLog(67): jetty-6.1.26 2023-07-11 14:20:43,105 INFO [Listener at localhost/35859] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/java.io.tmpdir/Jetty_localhost_39743_hdfs____.szosjq/webapp 2023-07-11 14:20:43,200 INFO [Listener at localhost/35859] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39743 2023-07-11 14:20:43,205 WARN [Listener at localhost/35859] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-11 14:20:43,205 WARN [Listener at localhost/35859] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-11 14:20:43,244 WARN [Listener at localhost/43147] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-11 14:20:43,255 WARN [Listener at localhost/43147] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-11 14:20:43,257 WARN [Listener at localhost/43147] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-11 14:20:43,258 INFO [Listener at localhost/43147] log.Slf4jLog(67): jetty-6.1.26 2023-07-11 14:20:43,262 INFO [Listener at localhost/43147] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/java.io.tmpdir/Jetty_localhost_33193_datanode____.sexf5a/webapp 2023-07-11 14:20:43,356 INFO [Listener at localhost/43147] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33193 2023-07-11 14:20:43,363 WARN [Listener at localhost/45445] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-11 14:20:43,453 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xecf798d5fad6a0b5: Processing first storage report for DS-cd205a8b-0c62-4681-a092-17d405a77c99 from datanode 41c70276-9c23-4ed1-80d1-224e00e225f4 2023-07-11 14:20:43,453 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xecf798d5fad6a0b5: from storage DS-cd205a8b-0c62-4681-a092-17d405a77c99 node DatanodeRegistration(127.0.0.1:33177, datanodeUuid=41c70276-9c23-4ed1-80d1-224e00e225f4, infoPort=33887, infoSecurePort=0, ipcPort=45445, storageInfo=lv=-57;cid=testClusterID;nsid=1373523690;c=1689085243061), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-11 14:20:43,453 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xecf798d5fad6a0b5: Processing first storage report for DS-5b4920d4-f799-49c4-931d-cbb106939f0a from datanode 41c70276-9c23-4ed1-80d1-224e00e225f4 2023-07-11 14:20:43,453 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xecf798d5fad6a0b5: from storage DS-5b4920d4-f799-49c4-931d-cbb106939f0a node DatanodeRegistration(127.0.0.1:33177, datanodeUuid=41c70276-9c23-4ed1-80d1-224e00e225f4, infoPort=33887, infoSecurePort=0, ipcPort=45445, storageInfo=lv=-57;cid=testClusterID;nsid=1373523690;c=1689085243061), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-11 14:20:43,468 DEBUG [Listener at localhost/45445] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa 2023-07-11 14:20:43,470 INFO [Listener at localhost/45445] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/cluster_ef34d284-81d2-8651-5312-c5ca43158945/zookeeper_0, clientPort=50076, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/cluster_ef34d284-81d2-8651-5312-c5ca43158945/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/cluster_ef34d284-81d2-8651-5312-c5ca43158945/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-11 14:20:43,472 INFO [Listener at localhost/45445] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=50076 2023-07-11 14:20:43,472 INFO [Listener at localhost/45445] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:43,473 INFO [Listener at localhost/45445] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:43,883 INFO [Listener at localhost/45445] util.FSUtils(471): Created version file at hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e with version=8 2023-07-11 14:20:43,883 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/hbase-staging 2023-07-11 14:20:43,885 INFO [Listener at localhost/45445] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-07-11 14:20:43,885 INFO [Listener at localhost/45445] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:43,885 INFO [Listener at localhost/45445] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:43,885 INFO [Listener at localhost/45445] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 14:20:43,885 INFO [Listener at localhost/45445] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:43,885 INFO [Listener at localhost/45445] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 14:20:43,885 INFO [Listener at localhost/45445] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 14:20:43,887 INFO [Listener at localhost/45445] ipc.NettyRpcServer(120): Bind to /172.31.2.10:45893 2023-07-11 14:20:43,887 INFO [Listener at localhost/45445] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:43,888 INFO [Listener at localhost/45445] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:43,889 INFO [Listener at localhost/45445] zookeeper.RecoverableZooKeeper(93): Process identifier=master:45893 connecting to ZooKeeper ensemble=127.0.0.1:50076 2023-07-11 14:20:43,895 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:458930x0, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 14:20:43,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:45893-0x10154b433ed0000 connected 2023-07-11 14:20:43,910 DEBUG [Listener at localhost/45445] zookeeper.ZKUtil(164): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:43,911 DEBUG [Listener at localhost/45445] zookeeper.ZKUtil(164): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:43,911 DEBUG [Listener at localhost/45445] zookeeper.ZKUtil(164): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 14:20:43,911 DEBUG [Listener at localhost/45445] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45893 2023-07-11 14:20:43,911 DEBUG [Listener at localhost/45445] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45893 2023-07-11 14:20:43,912 DEBUG [Listener at localhost/45445] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45893 2023-07-11 14:20:43,912 DEBUG [Listener at localhost/45445] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45893 2023-07-11 14:20:43,912 DEBUG [Listener at localhost/45445] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45893 2023-07-11 14:20:43,912 INFO [Listener at localhost/45445] master.HMaster(444): hbase.rootdir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e, hbase.cluster.distributed=false 2023-07-11 14:20:43,914 INFO [Listener at localhost/45445] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-07-11 14:20:43,914 INFO [Listener at localhost/45445] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:43,914 INFO [Listener at localhost/45445] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:43,914 INFO [Listener at localhost/45445] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 14:20:43,914 INFO [Listener at localhost/45445] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:43,914 INFO [Listener at localhost/45445] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 14:20:43,914 INFO [Listener at localhost/45445] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 14:20:43,917 INFO [Listener at localhost/45445] ipc.NettyRpcServer(120): Bind to /172.31.2.10:38655 2023-07-11 14:20:43,917 INFO [Listener at localhost/45445] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:43,918 INFO [Listener at localhost/45445] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:43,919 INFO [Listener at localhost/45445] zookeeper.RecoverableZooKeeper(93): Process identifier=master:38655 connecting to ZooKeeper ensemble=127.0.0.1:50076 2023-07-11 14:20:43,926 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:386550x0, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 14:20:43,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:38655-0x10154b433ed0001 connected 2023-07-11 14:20:43,930 DEBUG [Listener at localhost/45445] zookeeper.ZKUtil(164): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:43,930 DEBUG [Listener at localhost/45445] zookeeper.ZKUtil(164): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:43,930 DEBUG [Listener at localhost/45445] zookeeper.ZKUtil(164): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 14:20:43,931 DEBUG [Listener at localhost/45445] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38655 2023-07-11 14:20:43,932 DEBUG [Listener at localhost/45445] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38655 2023-07-11 14:20:43,932 DEBUG [Listener at localhost/45445] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38655 2023-07-11 14:20:43,932 DEBUG [Listener at localhost/45445] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38655 2023-07-11 14:20:43,932 DEBUG [Listener at localhost/45445] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38655 2023-07-11 14:20:43,933 INFO [Listener at localhost/45445] master.HMaster(444): hbase.rootdir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e, hbase.cluster.distributed=false 2023-07-11 14:20:43,945 INFO [Listener at localhost/45445] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-07-11 14:20:43,945 INFO [Listener at localhost/45445] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:43,945 INFO [Listener at localhost/45445] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:43,945 INFO [Listener at localhost/45445] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 14:20:43,945 INFO [Listener at localhost/45445] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:20:43,945 INFO [Listener at localhost/45445] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 14:20:43,945 INFO [Listener at localhost/45445] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 14:20:43,946 INFO [Listener at localhost/45445] ipc.NettyRpcServer(120): Bind to /172.31.2.10:33099 2023-07-11 14:20:43,946 INFO [Listener at localhost/45445] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-11 14:20:43,947 DEBUG [Listener at localhost/45445] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-11 14:20:43,947 INFO [Listener at localhost/45445] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:43,948 INFO [Listener at localhost/45445] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:43,949 INFO [Listener at localhost/45445] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33099 connecting to ZooKeeper ensemble=127.0.0.1:50076 2023-07-11 14:20:43,952 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): regionserver:330990x0, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 14:20:43,953 DEBUG [Listener at localhost/45445] zookeeper.ZKUtil(164): regionserver:330990x0, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:43,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:33099-0x10154b433ed0002 connected 2023-07-11 14:20:43,953 DEBUG [Listener at localhost/45445] zookeeper.ZKUtil(164): regionserver:33099-0x10154b433ed0002, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:43,954 DEBUG [Listener at localhost/45445] zookeeper.ZKUtil(164): regionserver:33099-0x10154b433ed0002, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 14:20:43,954 DEBUG [Listener at localhost/45445] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33099 2023-07-11 14:20:43,954 DEBUG [Listener at localhost/45445] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33099 2023-07-11 14:20:43,954 DEBUG [Listener at localhost/45445] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33099 2023-07-11 14:20:43,955 DEBUG [Listener at localhost/45445] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33099 2023-07-11 14:20:43,955 DEBUG [Listener at localhost/45445] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33099 2023-07-11 14:20:43,956 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,45893,1689085243884 2023-07-11 14:20:43,956 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,38655,1689085243913 2023-07-11 14:20:43,957 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:43,957 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:43,958 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,45893,1689085243884 2023-07-11 14:20:43,958 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,38655,1689085243913 2023-07-11 14:20:43,960 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): regionserver:33099-0x10154b433ed0002, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:43,960 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:43,960 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:43,960 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:43,960 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:43,961 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:43,961 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:43,962 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase9.apache.org,45893,1689085243884; waiting to become the next active master 2023-07-11 14:20:43,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:43,962 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,45893,1689085243884 from backup master directory 2023-07-11 14:20:43,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:43,963 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,45893,1689085243884 2023-07-11 14:20:43,963 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:43,963 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 14:20:43,963 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase9.apache.org,45893,1689085243884 2023-07-11 14:20:43,963 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:44,377 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/hbase.id with ID: 057f1d26-d323-454d-b27b-ec4242cea76b 2023-07-11 14:20:44,388 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:44,391 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:44,391 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:44,798 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x44c19f59 to 127.0.0.1:50076 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:20:44,801 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@277f8e43, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:20:44,801 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-11 14:20:44,802 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-11 14:20:44,802 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:20:44,803 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store-tmp 2023-07-11 14:20:45,210 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:45,210 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 14:20:45,210 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:45,210 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:45,210 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 14:20:45,210 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:45,210 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:45,210 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:45,211 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/WALs/jenkins-hbase9.apache.org,45893,1689085243884 2023-07-11 14:20:45,214 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C45893%2C1689085243884, suffix=, logDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/WALs/jenkins-hbase9.apache.org,45893,1689085243884, archiveDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/oldWALs, maxLogs=10 2023-07-11 14:20:45,232 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33177,DS-cd205a8b-0c62-4681-a092-17d405a77c99,DISK] 2023-07-11 14:20:45,234 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/WALs/jenkins-hbase9.apache.org,45893,1689085243884/jenkins-hbase9.apache.org%2C45893%2C1689085243884.1689085245214 2023-07-11 14:20:45,235 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33177,DS-cd205a8b-0c62-4681-a092-17d405a77c99,DISK]] 2023-07-11 14:20:45,235 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:45,235 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:45,235 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:45,235 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:45,236 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:45,238 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-11 14:20:45,239 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-11 14:20:45,239 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:45,240 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:45,241 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:45,244 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:45,246 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:45,246 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63471682, jitterRate=-0.054198235273361206}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 14:20:45,246 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:45,246 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-11 14:20:45,248 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:45,248 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:45,248 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:45,248 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:45,248 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-07-11 14:20:45,248 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-11 14:20:45,250 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-11 14:20:45,251 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-11 14:20:45,262 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-11 14:20:45,262 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-11 14:20:45,262 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-11 14:20:45,262 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-11 14:20:45,263 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-11 14:20:45,264 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:45,265 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:45,265 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-11 14:20:45,265 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-11 14:20:45,266 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-11 14:20:45,267 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:45,267 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:45,267 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:45,267 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): regionserver:33099-0x10154b433ed0002, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:45,267 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:45,267 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase9.apache.org,45893,1689085243884, sessionid=0x10154b433ed0000, setting cluster-up flag (Was=false) 2023-07-11 14:20:45,271 INFO [M:1;jenkins-hbase9:38655] regionserver.HRegionServer(951): ClusterId : 057f1d26-d323-454d-b27b-ec4242cea76b 2023-07-11 14:20:45,272 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:45,272 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:45,277 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-11 14:20:45,278 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,45893,1689085243884 2023-07-11 14:20:45,280 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:45,280 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:45,284 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-11 14:20:45,285 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,45893,1689085243884 2023-07-11 14:20:45,285 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.hbase-snapshot/.tmp 2023-07-11 14:20:45,287 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:45,287 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:45,288 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:45,288 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:45,288 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:45,288 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-07-11 14:20:45,288 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:45,288 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-11 14:20:45,288 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:45,288 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085275288 2023-07-11 14:20:45,288 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-11 14:20:45,289 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-11 14:20:45,289 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-11 14:20:45,289 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-11 14:20:45,289 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-11 14:20:45,289 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-11 14:20:45,289 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:45,290 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-11 14:20:45,290 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:45,290 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-11 14:20:45,290 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-11 14:20:45,290 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-11 14:20:45,290 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-11 14:20:45,290 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-11 14:20:45,291 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-11 14:20:45,294 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085245290,5,FailOnTimeoutGroup] 2023-07-11 14:20:45,295 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085245294,5,FailOnTimeoutGroup] 2023-07-11 14:20:45,295 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:45,295 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-11 14:20:45,295 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:45,295 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:45,358 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(951): ClusterId : 057f1d26-d323-454d-b27b-ec4242cea76b 2023-07-11 14:20:45,358 DEBUG [RS:0;jenkins-hbase9:33099] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-11 14:20:45,360 DEBUG [RS:0;jenkins-hbase9:33099] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-11 14:20:45,360 DEBUG [RS:0;jenkins-hbase9:33099] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-11 14:20:45,362 DEBUG [RS:0;jenkins-hbase9:33099] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-11 14:20:45,363 DEBUG [RS:0;jenkins-hbase9:33099] zookeeper.ReadOnlyZKClient(139): Connect 0x31cc45fc to 127.0.0.1:50076 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:20:45,367 DEBUG [RS:0;jenkins-hbase9:33099] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f0bbdb5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:20:45,368 DEBUG [RS:0;jenkins-hbase9:33099] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b342e8c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-11 14:20:45,377 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase9:33099 2023-07-11 14:20:45,377 INFO [RS:0;jenkins-hbase9:33099] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-11 14:20:45,377 INFO [RS:0;jenkins-hbase9:33099] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-11 14:20:45,377 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1022): About to register with Master. 2023-07-11 14:20:45,377 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase9.apache.org,45893,1689085243884 with isa=jenkins-hbase9.apache.org/172.31.2.10:33099, startcode=1689085243944 2023-07-11 14:20:45,377 DEBUG [RS:0;jenkins-hbase9:33099] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-11 14:20:45,379 INFO [RS-EventLoopGroup-12-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:60775, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=RegionServerStatusService 2023-07-11 14:20:45,380 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45893] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:45,381 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e 2023-07-11 14:20:45,381 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:43147 2023-07-11 14:20:45,381 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-11 14:20:45,382 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 14:20:45,382 DEBUG [RS:0;jenkins-hbase9:33099] zookeeper.ZKUtil(162): regionserver:33099-0x10154b433ed0002, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:45,383 WARN [RS:0;jenkins-hbase9:33099] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 14:20:45,383 INFO [RS:0;jenkins-hbase9:33099] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:20:45,383 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1948): logDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/WALs/jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:45,383 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,33099,1689085243944] 2023-07-11 14:20:45,386 DEBUG [RS:0;jenkins-hbase9:33099] zookeeper.ZKUtil(162): regionserver:33099-0x10154b433ed0002, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:45,387 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-11 14:20:45,387 INFO [RS:0;jenkins-hbase9:33099] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-11 14:20:45,389 INFO [RS:0;jenkins-hbase9:33099] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-11 14:20:45,389 INFO [RS:0;jenkins-hbase9:33099] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-11 14:20:45,389 INFO [RS:0;jenkins-hbase9:33099] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:45,390 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-11 14:20:45,391 INFO [RS:0;jenkins-hbase9:33099] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:45,391 DEBUG [RS:0;jenkins-hbase9:33099] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:45,391 DEBUG [RS:0;jenkins-hbase9:33099] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:45,391 DEBUG [RS:0;jenkins-hbase9:33099] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:45,391 DEBUG [RS:0;jenkins-hbase9:33099] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:45,391 DEBUG [RS:0;jenkins-hbase9:33099] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:45,391 DEBUG [RS:0;jenkins-hbase9:33099] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-11 14:20:45,391 DEBUG [RS:0;jenkins-hbase9:33099] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:45,391 DEBUG [RS:0;jenkins-hbase9:33099] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:45,391 DEBUG [RS:0;jenkins-hbase9:33099] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:45,391 DEBUG [RS:0;jenkins-hbase9:33099] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:45,392 INFO [RS:0;jenkins-hbase9:33099] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:45,392 INFO [RS:0;jenkins-hbase9:33099] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:45,392 INFO [RS:0;jenkins-hbase9:33099] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:45,403 INFO [RS:0;jenkins-hbase9:33099] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-11 14:20:45,403 INFO [RS:0;jenkins-hbase9:33099] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,33099,1689085243944-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:45,414 INFO [RS:0;jenkins-hbase9:33099] regionserver.Replication(203): jenkins-hbase9.apache.org,33099,1689085243944 started 2023-07-11 14:20:45,414 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1637): Serving as jenkins-hbase9.apache.org,33099,1689085243944, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:33099, sessionid=0x10154b433ed0002 2023-07-11 14:20:45,414 DEBUG [RS:0;jenkins-hbase9:33099] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-11 14:20:45,414 DEBUG [RS:0;jenkins-hbase9:33099] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:45,414 DEBUG [RS:0;jenkins-hbase9:33099] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33099,1689085243944' 2023-07-11 14:20:45,414 DEBUG [RS:0;jenkins-hbase9:33099] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-11 14:20:45,415 DEBUG [RS:0;jenkins-hbase9:33099] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-11 14:20:45,415 DEBUG [RS:0;jenkins-hbase9:33099] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-11 14:20:45,415 DEBUG [RS:0;jenkins-hbase9:33099] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-11 14:20:45,415 DEBUG [RS:0;jenkins-hbase9:33099] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:45,415 DEBUG [RS:0;jenkins-hbase9:33099] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,33099,1689085243944' 2023-07-11 14:20:45,415 DEBUG [RS:0;jenkins-hbase9:33099] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-11 14:20:45,416 DEBUG [RS:0;jenkins-hbase9:33099] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-11 14:20:45,416 DEBUG [RS:0;jenkins-hbase9:33099] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-11 14:20:45,416 INFO [RS:0;jenkins-hbase9:33099] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-11 14:20:45,416 INFO [RS:0;jenkins-hbase9:33099] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-11 14:20:45,518 INFO [RS:0;jenkins-hbase9:33099] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C33099%2C1689085243944, suffix=, logDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/WALs/jenkins-hbase9.apache.org,33099,1689085243944, archiveDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/oldWALs, maxLogs=32 2023-07-11 14:20:45,535 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33177,DS-cd205a8b-0c62-4681-a092-17d405a77c99,DISK] 2023-07-11 14:20:45,537 INFO [RS:0;jenkins-hbase9:33099] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/WALs/jenkins-hbase9.apache.org,33099,1689085243944/jenkins-hbase9.apache.org%2C33099%2C1689085243944.1689085245519 2023-07-11 14:20:45,537 DEBUG [RS:0;jenkins-hbase9:33099] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33177,DS-cd205a8b-0c62-4681-a092-17d405a77c99,DISK]] 2023-07-11 14:20:45,700 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:45,701 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:45,701 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e 2023-07-11 14:20:46,107 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:46,108 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-11 14:20:46,109 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/info 2023-07-11 14:20:46,110 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-11 14:20:46,110 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:46,111 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-11 14:20:46,112 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/rep_barrier 2023-07-11 14:20:46,112 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-11 14:20:46,112 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:46,112 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-11 14:20:46,114 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/table 2023-07-11 14:20:46,114 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-11 14:20:46,114 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:46,115 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740 2023-07-11 14:20:46,115 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740 2023-07-11 14:20:46,117 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-11 14:20:46,118 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-11 14:20:46,120 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:46,120 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66798537, jitterRate=-0.004624232649803162}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-11 14:20:46,121 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-11 14:20:46,121 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-11 14:20:46,121 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-11 14:20:46,121 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-11 14:20:46,121 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-11 14:20:46,121 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-11 14:20:46,121 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-11 14:20:46,121 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-11 14:20:46,122 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:46,122 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-11 14:20:46,122 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-11 14:20:46,124 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-11 14:20:46,124 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-11 14:20:46,275 DEBUG [jenkins-hbase9:45893] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-11 14:20:46,275 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,33099,1689085243944, state=OPENING 2023-07-11 14:20:46,277 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-11 14:20:46,278 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:46,278 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:46,279 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:20:46,279 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,33099,1689085243944}] 2023-07-11 14:20:46,279 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:20:46,431 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:46,432 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-11 14:20:46,434 INFO [RS-EventLoopGroup-14-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:40700, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-11 14:20:46,437 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-11 14:20:46,437 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:20:46,439 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C33099%2C1689085243944.meta, suffix=.meta, logDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/WALs/jenkins-hbase9.apache.org,33099,1689085243944, archiveDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/oldWALs, maxLogs=32 2023-07-11 14:20:46,454 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33177,DS-cd205a8b-0c62-4681-a092-17d405a77c99,DISK] 2023-07-11 14:20:46,455 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/WALs/jenkins-hbase9.apache.org,33099,1689085243944/jenkins-hbase9.apache.org%2C33099%2C1689085243944.meta.1689085246439.meta 2023-07-11 14:20:46,456 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33177,DS-cd205a8b-0c62-4681-a092-17d405a77c99,DISK]] 2023-07-11 14:20:46,456 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:46,456 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-11 14:20:46,456 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-11 14:20:46,456 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-11 14:20:46,456 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-11 14:20:46,456 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:46,456 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-11 14:20:46,456 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-11 14:20:46,458 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-11 14:20:46,458 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/info 2023-07-11 14:20:46,459 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/info 2023-07-11 14:20:46,459 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-11 14:20:46,459 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:46,460 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-11 14:20:46,460 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/rep_barrier 2023-07-11 14:20:46,461 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/rep_barrier 2023-07-11 14:20:46,461 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-11 14:20:46,462 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:46,462 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-11 14:20:46,463 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/table 2023-07-11 14:20:46,463 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/table 2023-07-11 14:20:46,463 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-11 14:20:46,464 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:46,464 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740 2023-07-11 14:20:46,466 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740 2023-07-11 14:20:46,467 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-11 14:20:46,468 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-11 14:20:46,469 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68286628, jitterRate=0.01755005121231079}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-11 14:20:46,469 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-11 14:20:46,470 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689085246431 2023-07-11 14:20:46,473 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-11 14:20:46,473 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-11 14:20:46,473 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,33099,1689085243944, state=OPEN 2023-07-11 14:20:46,480 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-11 14:20:46,480 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-11 14:20:46,480 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:20:46,480 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:20:46,482 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-11 14:20:46,482 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,33099,1689085243944 in 201 msec 2023-07-11 14:20:46,483 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-11 14:20:46,484 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 360 msec 2023-07-11 14:20:46,485 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1980 sec 2023-07-11 14:20:46,485 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689085246485, completionTime=-1 2023-07-11 14:20:46,485 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-11 14:20:46,485 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:46,488 DEBUG [hconnection-0x757cbe02-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 14:20:46,489 INFO [RS-EventLoopGroup-14-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:40712, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 14:20:46,491 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:46,491 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085306491 2023-07-11 14:20:46,491 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085366491 2023-07-11 14:20:46,491 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-07-11 14:20:46,496 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,45893,1689085243884-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:46,496 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,45893,1689085243884-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:46,496 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,45893,1689085243884-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:46,496 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:45893, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:46,496 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:46,497 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-11 14:20:46,497 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-11 14:20:46,498 DEBUG [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-11 14:20:46,498 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:46,499 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-11 14:20:46,500 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-11 14:20:46,501 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/hbase/namespace/d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:46,502 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/hbase/namespace/d0027ffc5e8dfd7f30f021f0fdb84921 empty. 2023-07-11 14:20:46,502 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/hbase/namespace/d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:46,502 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-11 14:20:46,911 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:46,912 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => d0027ffc5e8dfd7f30f021f0fdb84921, NAME => 'hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp 2023-07-11 14:20:47,073 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-11 14:20:47,319 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:47,319 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing d0027ffc5e8dfd7f30f021f0fdb84921, disabling compactions & flushes 2023-07-11 14:20:47,319 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921. 2023-07-11 14:20:47,319 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921. 2023-07-11 14:20:47,319 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921. after waiting 0 ms 2023-07-11 14:20:47,319 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921. 2023-07-11 14:20:47,319 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921. 2023-07-11 14:20:47,320 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for d0027ffc5e8dfd7f30f021f0fdb84921: 2023-07-11 14:20:47,322 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-11 14:20:47,323 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689085247323"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085247323"}]},"ts":"1689085247323"} 2023-07-11 14:20:47,325 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-11 14:20:47,326 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-11 14:20:47,326 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085247326"}]},"ts":"1689085247326"} 2023-07-11 14:20:47,328 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-11 14:20:47,332 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d0027ffc5e8dfd7f30f021f0fdb84921, ASSIGN}] 2023-07-11 14:20:47,333 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d0027ffc5e8dfd7f30f021f0fdb84921, ASSIGN 2023-07-11 14:20:47,334 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=d0027ffc5e8dfd7f30f021f0fdb84921, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,33099,1689085243944; forceNewPlan=false, retain=false 2023-07-11 14:20:47,485 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=d0027ffc5e8dfd7f30f021f0fdb84921, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:47,485 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689085247485"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085247485"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085247485"}]},"ts":"1689085247485"} 2023-07-11 14:20:47,487 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure d0027ffc5e8dfd7f30f021f0fdb84921, server=jenkins-hbase9.apache.org,33099,1689085243944}] 2023-07-11 14:20:47,643 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921. 2023-07-11 14:20:47,643 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d0027ffc5e8dfd7f30f021f0fdb84921, NAME => 'hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:47,644 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:47,644 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:47,644 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:47,644 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:47,645 INFO [StoreOpener-d0027ffc5e8dfd7f30f021f0fdb84921-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:47,646 DEBUG [StoreOpener-d0027ffc5e8dfd7f30f021f0fdb84921-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/namespace/d0027ffc5e8dfd7f30f021f0fdb84921/info 2023-07-11 14:20:47,646 DEBUG [StoreOpener-d0027ffc5e8dfd7f30f021f0fdb84921-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/namespace/d0027ffc5e8dfd7f30f021f0fdb84921/info 2023-07-11 14:20:47,647 INFO [StoreOpener-d0027ffc5e8dfd7f30f021f0fdb84921-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d0027ffc5e8dfd7f30f021f0fdb84921 columnFamilyName info 2023-07-11 14:20:47,648 INFO [StoreOpener-d0027ffc5e8dfd7f30f021f0fdb84921-1] regionserver.HStore(310): Store=d0027ffc5e8dfd7f30f021f0fdb84921/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:47,648 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/namespace/d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:47,649 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/namespace/d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:47,651 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:47,653 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/namespace/d0027ffc5e8dfd7f30f021f0fdb84921/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:47,653 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened d0027ffc5e8dfd7f30f021f0fdb84921; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62655514, jitterRate=-0.06636008620262146}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 14:20:47,653 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for d0027ffc5e8dfd7f30f021f0fdb84921: 2023-07-11 14:20:47,654 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921., pid=6, masterSystemTime=1689085247639 2023-07-11 14:20:47,656 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921. 2023-07-11 14:20:47,656 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921. 2023-07-11 14:20:47,657 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=d0027ffc5e8dfd7f30f021f0fdb84921, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:47,657 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689085247657"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085247657"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085247657"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085247657"}]},"ts":"1689085247657"} 2023-07-11 14:20:47,660 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-11 14:20:47,660 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure d0027ffc5e8dfd7f30f021f0fdb84921, server=jenkins-hbase9.apache.org,33099,1689085243944 in 172 msec 2023-07-11 14:20:47,662 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-11 14:20:47,662 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=d0027ffc5e8dfd7f30f021f0fdb84921, ASSIGN in 328 msec 2023-07-11 14:20:47,663 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-11 14:20:47,663 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085247663"}]},"ts":"1689085247663"} 2023-07-11 14:20:47,664 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-11 14:20:47,666 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-11 14:20:47,668 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.1690 sec 2023-07-11 14:20:47,699 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-11 14:20:47,704 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:47,704 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-11 14:20:47,704 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:47,708 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:47,715 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-11 14:20:47,718 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 10 msec 2023-07-11 14:20:47,719 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:47,726 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-11 14:20:47,729 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 9 msec 2023-07-11 14:20:47,733 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-11 14:20:47,736 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-11 14:20:47,736 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.773sec 2023-07-11 14:20:47,736 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-11 14:20:47,736 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-11 14:20:47,736 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-11 14:20:47,736 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,45893,1689085243884-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-11 14:20:47,736 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,45893,1689085243884-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-11 14:20:47,738 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-11 14:20:47,759 DEBUG [Listener at localhost/45445] zookeeper.ReadOnlyZKClient(139): Connect 0x0a665a39 to 127.0.0.1:50076 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:20:47,764 DEBUG [Listener at localhost/45445] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@106fd1f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:20:47,766 DEBUG [hconnection-0x445f98c5-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 14:20:47,769 INFO [RS-EventLoopGroup-14-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:40718, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 14:20:47,770 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase9.apache.org,45893,1689085243884 2023-07-11 14:20:47,770 WARN [Listener at localhost/45445] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-11 14:20:47,773 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-11 14:20:47,774 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-11 14:20:47,775 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-11 14:20:47,779 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7 2023-07-11 14:20:47,779 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34 2023-07-11 14:20:47,779 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745 2023-07-11 14:20:47,779 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249 2023-07-11 14:20:47,780 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7 empty. 2023-07-11 14:20:47,780 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249 empty. 2023-07-11 14:20:47,781 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745 empty. 2023-07-11 14:20:47,781 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34 empty. 2023-07-11 14:20:47,781 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249 2023-07-11 14:20:47,781 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34 2023-07-11 14:20:47,781 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7 2023-07-11 14:20:47,782 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745 2023-07-11 14:20:47,782 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTruncateWithFailoverAtStep4 regions 2023-07-11 14:20:47,794 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:47,847 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(7675): creating {ENCODED => 15ef30302259990af4c93ea5956e8be7, NAME => 'testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7.', STARTKEY => '', ENDKEY => 'a'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp 2023-07-11 14:20:47,848 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(7675): creating {ENCODED => c887ddcecdd2e29691c3631bd0b10745, NAME => 'testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745.', STARTKEY => 'b', ENDKEY => 'c'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp 2023-07-11 14:20:47,848 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(7675): creating {ENCODED => deb98f3ca0218078c17361c806c6f249, NAME => 'testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249.', STARTKEY => 'a', ENDKEY => 'b'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp 2023-07-11 14:20:47,862 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:47,862 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1604): Closing c887ddcecdd2e29691c3631bd0b10745, disabling compactions & flushes 2023-07-11 14:20:47,862 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745. 2023-07-11 14:20:47,862 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745. 2023-07-11 14:20:47,862 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745. after waiting 0 ms 2023-07-11 14:20:47,862 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745. 2023-07-11 14:20:47,862 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745. 2023-07-11 14:20:47,862 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1558): Region close journal for c887ddcecdd2e29691c3631bd0b10745: 2023-07-11 14:20:47,863 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(7675): creating {ENCODED => 6c361c95cb323d45992c0bfbbe2c6f34, NAME => 'testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34.', STARTKEY => 'c', ENDKEY => ''}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp 2023-07-11 14:20:48,262 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:48,262 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:48,262 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1604): Closing deb98f3ca0218078c17361c806c6f249, disabling compactions & flushes 2023-07-11 14:20:48,262 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1604): Closing 15ef30302259990af4c93ea5956e8be7, disabling compactions & flushes 2023-07-11 14:20:48,262 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249. 2023-07-11 14:20:48,262 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7. 2023-07-11 14:20:48,262 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7. 2023-07-11 14:20:48,262 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249. 2023-07-11 14:20:48,262 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7. after waiting 0 ms 2023-07-11 14:20:48,262 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7. 2023-07-11 14:20:48,262 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249. after waiting 0 ms 2023-07-11 14:20:48,262 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249. 2023-07-11 14:20:48,262 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7. 2023-07-11 14:20:48,262 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249. 2023-07-11 14:20:48,263 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1558): Region close journal for deb98f3ca0218078c17361c806c6f249: 2023-07-11 14:20:48,263 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1558): Region close journal for 15ef30302259990af4c93ea5956e8be7: 2023-07-11 14:20:48,268 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:48,269 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1604): Closing 6c361c95cb323d45992c0bfbbe2c6f34, disabling compactions & flushes 2023-07-11 14:20:48,269 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34. 2023-07-11 14:20:48,269 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34. 2023-07-11 14:20:48,269 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34. after waiting 0 ms 2023-07-11 14:20:48,269 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34. 2023-07-11 14:20:48,269 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34. 2023-07-11 14:20:48,269 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1558): Region close journal for 6c361c95cb323d45992c0bfbbe2c6f34: 2023-07-11 14:20:48,271 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_ADD_TO_META 2023-07-11 14:20:48,272 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085248272"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085248272"}]},"ts":"1689085248272"} 2023-07-11 14:20:48,272 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085248272"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085248272"}]},"ts":"1689085248272"} 2023-07-11 14:20:48,272 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085248272"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085248272"}]},"ts":"1689085248272"} 2023-07-11 14:20:48,272 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085248272"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085248272"}]},"ts":"1689085248272"} 2023-07-11 14:20:48,275 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 4 regions to meta. 2023-07-11 14:20:48,276 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-11 14:20:48,276 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085248276"}]},"ts":"1689085248276"} 2023-07-11 14:20:48,277 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=ENABLING in hbase:meta 2023-07-11 14:20:48,285 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=15ef30302259990af4c93ea5956e8be7, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=deb98f3ca0218078c17361c806c6f249, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=c887ddcecdd2e29691c3631bd0b10745, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=6c361c95cb323d45992c0bfbbe2c6f34, ASSIGN}] 2023-07-11 14:20:48,286 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=6c361c95cb323d45992c0bfbbe2c6f34, ASSIGN 2023-07-11 14:20:48,287 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=6c361c95cb323d45992c0bfbbe2c6f34, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,33099,1689085243944; forceNewPlan=false, retain=false 2023-07-11 14:20:48,287 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=c887ddcecdd2e29691c3631bd0b10745, ASSIGN 2023-07-11 14:20:48,288 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=c887ddcecdd2e29691c3631bd0b10745, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,33099,1689085243944; forceNewPlan=false, retain=false 2023-07-11 14:20:48,289 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=deb98f3ca0218078c17361c806c6f249, ASSIGN 2023-07-11 14:20:48,289 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=deb98f3ca0218078c17361c806c6f249, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,33099,1689085243944; forceNewPlan=false, retain=false 2023-07-11 14:20:48,290 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=15ef30302259990af4c93ea5956e8be7, ASSIGN 2023-07-11 14:20:48,291 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=15ef30302259990af4c93ea5956e8be7, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,33099,1689085243944; forceNewPlan=false, retain=false 2023-07-11 14:20:48,439 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=6c361c95cb323d45992c0bfbbe2c6f34, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:48,439 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085248439"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085248439"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085248439"}]},"ts":"1689085248439"} 2023-07-11 14:20:48,441 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; OpenRegionProcedure 6c361c95cb323d45992c0bfbbe2c6f34, server=jenkins-hbase9.apache.org,33099,1689085243944}] 2023-07-11 14:20:48,444 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=c887ddcecdd2e29691c3631bd0b10745, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:48,444 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085248444"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085248444"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085248444"}]},"ts":"1689085248444"} 2023-07-11 14:20:48,446 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE; OpenRegionProcedure c887ddcecdd2e29691c3631bd0b10745, server=jenkins-hbase9.apache.org,33099,1689085243944}] 2023-07-11 14:20:48,448 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=15ef30302259990af4c93ea5956e8be7, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:48,448 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085248448"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085248448"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085248448"}]},"ts":"1689085248448"} 2023-07-11 14:20:48,449 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=10, state=RUNNABLE; OpenRegionProcedure 15ef30302259990af4c93ea5956e8be7, server=jenkins-hbase9.apache.org,33099,1689085243944}] 2023-07-11 14:20:48,452 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=deb98f3ca0218078c17361c806c6f249, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:48,452 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085248452"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085248452"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085248452"}]},"ts":"1689085248452"} 2023-07-11 14:20:48,453 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=11, state=RUNNABLE; OpenRegionProcedure deb98f3ca0218078c17361c806c6f249, server=jenkins-hbase9.apache.org,33099,1689085243944}] 2023-07-11 14:20:48,597 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7. 2023-07-11 14:20:48,597 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 15ef30302259990af4c93ea5956e8be7, NAME => 'testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7.', STARTKEY => '', ENDKEY => 'a'} 2023-07-11 14:20:48,597 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 15ef30302259990af4c93ea5956e8be7 2023-07-11 14:20:48,597 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:48,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 15ef30302259990af4c93ea5956e8be7 2023-07-11 14:20:48,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 15ef30302259990af4c93ea5956e8be7 2023-07-11 14:20:48,599 INFO [StoreOpener-15ef30302259990af4c93ea5956e8be7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 15ef30302259990af4c93ea5956e8be7 2023-07-11 14:20:48,600 DEBUG [StoreOpener-15ef30302259990af4c93ea5956e8be7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/f1 2023-07-11 14:20:48,600 DEBUG [StoreOpener-15ef30302259990af4c93ea5956e8be7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/f1 2023-07-11 14:20:48,600 INFO [StoreOpener-15ef30302259990af4c93ea5956e8be7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 15ef30302259990af4c93ea5956e8be7 columnFamilyName f1 2023-07-11 14:20:48,601 INFO [StoreOpener-15ef30302259990af4c93ea5956e8be7-1] regionserver.HStore(310): Store=15ef30302259990af4c93ea5956e8be7/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:48,601 INFO [StoreOpener-15ef30302259990af4c93ea5956e8be7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 15ef30302259990af4c93ea5956e8be7 2023-07-11 14:20:48,602 DEBUG [StoreOpener-15ef30302259990af4c93ea5956e8be7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/f2 2023-07-11 14:20:48,602 DEBUG [StoreOpener-15ef30302259990af4c93ea5956e8be7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/f2 2023-07-11 14:20:48,603 INFO [StoreOpener-15ef30302259990af4c93ea5956e8be7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 15ef30302259990af4c93ea5956e8be7 columnFamilyName f2 2023-07-11 14:20:48,603 INFO [StoreOpener-15ef30302259990af4c93ea5956e8be7-1] regionserver.HStore(310): Store=15ef30302259990af4c93ea5956e8be7/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:48,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7 2023-07-11 14:20:48,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7 2023-07-11 14:20:48,606 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:20:48,607 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 15ef30302259990af4c93ea5956e8be7 2023-07-11 14:20:48,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:48,609 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 15ef30302259990af4c93ea5956e8be7; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63784111, jitterRate=-0.04954268038272858}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:20:48,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 15ef30302259990af4c93ea5956e8be7: 2023-07-11 14:20:48,610 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7., pid=16, masterSystemTime=1689085248593 2023-07-11 14:20:48,612 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7. 2023-07-11 14:20:48,612 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7. 2023-07-11 14:20:48,612 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34. 2023-07-11 14:20:48,612 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6c361c95cb323d45992c0bfbbe2c6f34, NAME => 'testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34.', STARTKEY => 'c', ENDKEY => ''} 2023-07-11 14:20:48,612 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=15ef30302259990af4c93ea5956e8be7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:48,612 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 6c361c95cb323d45992c0bfbbe2c6f34 2023-07-11 14:20:48,612 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085248612"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085248612"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085248612"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085248612"}]},"ts":"1689085248612"} 2023-07-11 14:20:48,612 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:48,612 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 6c361c95cb323d45992c0bfbbe2c6f34 2023-07-11 14:20:48,612 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 6c361c95cb323d45992c0bfbbe2c6f34 2023-07-11 14:20:48,614 INFO [StoreOpener-6c361c95cb323d45992c0bfbbe2c6f34-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 6c361c95cb323d45992c0bfbbe2c6f34 2023-07-11 14:20:48,615 DEBUG [StoreOpener-6c361c95cb323d45992c0bfbbe2c6f34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/f1 2023-07-11 14:20:48,615 DEBUG [StoreOpener-6c361c95cb323d45992c0bfbbe2c6f34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/f1 2023-07-11 14:20:48,615 INFO [StoreOpener-6c361c95cb323d45992c0bfbbe2c6f34-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6c361c95cb323d45992c0bfbbe2c6f34 columnFamilyName f1 2023-07-11 14:20:48,616 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=10 2023-07-11 14:20:48,616 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=10, state=SUCCESS; OpenRegionProcedure 15ef30302259990af4c93ea5956e8be7, server=jenkins-hbase9.apache.org,33099,1689085243944 in 165 msec 2023-07-11 14:20:48,616 INFO [StoreOpener-6c361c95cb323d45992c0bfbbe2c6f34-1] regionserver.HStore(310): Store=6c361c95cb323d45992c0bfbbe2c6f34/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:48,616 INFO [StoreOpener-6c361c95cb323d45992c0bfbbe2c6f34-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 6c361c95cb323d45992c0bfbbe2c6f34 2023-07-11 14:20:48,617 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=15ef30302259990af4c93ea5956e8be7, ASSIGN in 332 msec 2023-07-11 14:20:48,617 DEBUG [StoreOpener-6c361c95cb323d45992c0bfbbe2c6f34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/f2 2023-07-11 14:20:48,617 DEBUG [StoreOpener-6c361c95cb323d45992c0bfbbe2c6f34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/f2 2023-07-11 14:20:48,618 INFO [StoreOpener-6c361c95cb323d45992c0bfbbe2c6f34-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6c361c95cb323d45992c0bfbbe2c6f34 columnFamilyName f2 2023-07-11 14:20:48,618 INFO [StoreOpener-6c361c95cb323d45992c0bfbbe2c6f34-1] regionserver.HStore(310): Store=6c361c95cb323d45992c0bfbbe2c6f34/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:48,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34 2023-07-11 14:20:48,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34 2023-07-11 14:20:48,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:20:48,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 6c361c95cb323d45992c0bfbbe2c6f34 2023-07-11 14:20:48,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:48,623 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 6c361c95cb323d45992c0bfbbe2c6f34; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59718346, jitterRate=-0.11012730002403259}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:20:48,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 6c361c95cb323d45992c0bfbbe2c6f34: 2023-07-11 14:20:48,624 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34., pid=14, masterSystemTime=1689085248593 2023-07-11 14:20:48,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34. 2023-07-11 14:20:48,625 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34. 2023-07-11 14:20:48,625 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249. 2023-07-11 14:20:48,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => deb98f3ca0218078c17361c806c6f249, NAME => 'testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249.', STARTKEY => 'a', ENDKEY => 'b'} 2023-07-11 14:20:48,626 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=6c361c95cb323d45992c0bfbbe2c6f34, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:48,626 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 deb98f3ca0218078c17361c806c6f249 2023-07-11 14:20:48,626 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:48,626 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085248626"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085248626"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085248626"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085248626"}]},"ts":"1689085248626"} 2023-07-11 14:20:48,626 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for deb98f3ca0218078c17361c806c6f249 2023-07-11 14:20:48,626 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for deb98f3ca0218078c17361c806c6f249 2023-07-11 14:20:48,627 INFO [StoreOpener-deb98f3ca0218078c17361c806c6f249-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region deb98f3ca0218078c17361c806c6f249 2023-07-11 14:20:48,628 DEBUG [StoreOpener-deb98f3ca0218078c17361c806c6f249-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/f1 2023-07-11 14:20:48,628 DEBUG [StoreOpener-deb98f3ca0218078c17361c806c6f249-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/f1 2023-07-11 14:20:48,628 INFO [StoreOpener-deb98f3ca0218078c17361c806c6f249-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region deb98f3ca0218078c17361c806c6f249 columnFamilyName f1 2023-07-11 14:20:48,629 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-11 14:20:48,629 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; OpenRegionProcedure 6c361c95cb323d45992c0bfbbe2c6f34, server=jenkins-hbase9.apache.org,33099,1689085243944 in 186 msec 2023-07-11 14:20:48,629 INFO [StoreOpener-deb98f3ca0218078c17361c806c6f249-1] regionserver.HStore(310): Store=deb98f3ca0218078c17361c806c6f249/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:48,629 INFO [StoreOpener-deb98f3ca0218078c17361c806c6f249-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region deb98f3ca0218078c17361c806c6f249 2023-07-11 14:20:48,630 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=6c361c95cb323d45992c0bfbbe2c6f34, ASSIGN in 345 msec 2023-07-11 14:20:48,630 DEBUG [StoreOpener-deb98f3ca0218078c17361c806c6f249-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/f2 2023-07-11 14:20:48,630 DEBUG [StoreOpener-deb98f3ca0218078c17361c806c6f249-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/f2 2023-07-11 14:20:48,630 INFO [StoreOpener-deb98f3ca0218078c17361c806c6f249-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region deb98f3ca0218078c17361c806c6f249 columnFamilyName f2 2023-07-11 14:20:48,631 INFO [StoreOpener-deb98f3ca0218078c17361c806c6f249-1] regionserver.HStore(310): Store=deb98f3ca0218078c17361c806c6f249/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:48,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249 2023-07-11 14:20:48,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249 2023-07-11 14:20:48,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:20:48,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for deb98f3ca0218078c17361c806c6f249 2023-07-11 14:20:48,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:48,636 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened deb98f3ca0218078c17361c806c6f249; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75362093, jitterRate=0.1229826956987381}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:20:48,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for deb98f3ca0218078c17361c806c6f249: 2023-07-11 14:20:48,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249., pid=17, masterSystemTime=1689085248593 2023-07-11 14:20:48,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249. 2023-07-11 14:20:48,638 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249. 2023-07-11 14:20:48,638 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745. 2023-07-11 14:20:48,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c887ddcecdd2e29691c3631bd0b10745, NAME => 'testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745.', STARTKEY => 'b', ENDKEY => 'c'} 2023-07-11 14:20:48,638 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=deb98f3ca0218078c17361c806c6f249, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:48,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 c887ddcecdd2e29691c3631bd0b10745 2023-07-11 14:20:48,639 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085248638"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085248638"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085248638"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085248638"}]},"ts":"1689085248638"} 2023-07-11 14:20:48,639 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:48,639 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for c887ddcecdd2e29691c3631bd0b10745 2023-07-11 14:20:48,639 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for c887ddcecdd2e29691c3631bd0b10745 2023-07-11 14:20:48,640 INFO [StoreOpener-c887ddcecdd2e29691c3631bd0b10745-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region c887ddcecdd2e29691c3631bd0b10745 2023-07-11 14:20:48,641 DEBUG [StoreOpener-c887ddcecdd2e29691c3631bd0b10745-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/f1 2023-07-11 14:20:48,641 DEBUG [StoreOpener-c887ddcecdd2e29691c3631bd0b10745-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/f1 2023-07-11 14:20:48,641 INFO [StoreOpener-c887ddcecdd2e29691c3631bd0b10745-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c887ddcecdd2e29691c3631bd0b10745 columnFamilyName f1 2023-07-11 14:20:48,641 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=11 2023-07-11 14:20:48,641 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=11, state=SUCCESS; OpenRegionProcedure deb98f3ca0218078c17361c806c6f249, server=jenkins-hbase9.apache.org,33099,1689085243944 in 187 msec 2023-07-11 14:20:48,642 INFO [StoreOpener-c887ddcecdd2e29691c3631bd0b10745-1] regionserver.HStore(310): Store=c887ddcecdd2e29691c3631bd0b10745/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:48,642 INFO [StoreOpener-c887ddcecdd2e29691c3631bd0b10745-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region c887ddcecdd2e29691c3631bd0b10745 2023-07-11 14:20:48,642 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=deb98f3ca0218078c17361c806c6f249, ASSIGN in 357 msec 2023-07-11 14:20:48,643 DEBUG [StoreOpener-c887ddcecdd2e29691c3631bd0b10745-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/f2 2023-07-11 14:20:48,643 DEBUG [StoreOpener-c887ddcecdd2e29691c3631bd0b10745-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/f2 2023-07-11 14:20:48,643 INFO [StoreOpener-c887ddcecdd2e29691c3631bd0b10745-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c887ddcecdd2e29691c3631bd0b10745 columnFamilyName f2 2023-07-11 14:20:48,644 INFO [StoreOpener-c887ddcecdd2e29691c3631bd0b10745-1] regionserver.HStore(310): Store=c887ddcecdd2e29691c3631bd0b10745/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:48,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745 2023-07-11 14:20:48,645 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745 2023-07-11 14:20:48,646 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:20:48,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for c887ddcecdd2e29691c3631bd0b10745 2023-07-11 14:20:48,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:48,649 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened c887ddcecdd2e29691c3631bd0b10745; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73361431, jitterRate=0.09317050874233246}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:20:48,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for c887ddcecdd2e29691c3631bd0b10745: 2023-07-11 14:20:48,650 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745., pid=15, masterSystemTime=1689085248593 2023-07-11 14:20:48,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745. 2023-07-11 14:20:48,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745. 2023-07-11 14:20:48,651 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=c887ddcecdd2e29691c3631bd0b10745, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:48,652 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085248651"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085248651"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085248651"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085248651"}]},"ts":"1689085248651"} 2023-07-11 14:20:48,654 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-11 14:20:48,654 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; OpenRegionProcedure c887ddcecdd2e29691c3631bd0b10745, server=jenkins-hbase9.apache.org,33099,1689085243944 in 207 msec 2023-07-11 14:20:48,656 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=12, resume processing ppid=9 2023-07-11 14:20:48,656 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=c887ddcecdd2e29691c3631bd0b10745, ASSIGN in 371 msec 2023-07-11 14:20:48,657 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-11 14:20:48,657 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085248657"}]},"ts":"1689085248657"} 2023-07-11 14:20:48,659 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=ENABLED in hbase:meta 2023-07-11 14:20:48,661 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_POST_OPERATION 2023-07-11 14:20:48,662 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 in 889 msec 2023-07-11 14:20:48,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33099] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7. with WAL disabled. Data may be lost in the event of a crash. 2023-07-11 14:20:48,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33099] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249. with WAL disabled. Data may be lost in the event of a crash. 2023-07-11 14:20:48,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33099] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745. with WAL disabled. Data may be lost in the event of a crash. 2023-07-11 14:20:48,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33099] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34. with WAL disabled. Data may be lost in the event of a crash. 2023-07-11 14:20:48,809 DEBUG [Listener at localhost/45445] client.ClientScanner(240): Advancing internal scanner to startKey at 'a', inclusive 2023-07-11 14:20:48,811 DEBUG [Listener at localhost/45445] client.ClientScanner(240): Advancing internal scanner to startKey at 'b', inclusive 2023-07-11 14:20:48,814 DEBUG [Listener at localhost/45445] client.ClientScanner(240): Advancing internal scanner to startKey at 'c', inclusive 2023-07-11 14:20:48,816 DEBUG [Listener at localhost/45445] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-11 14:20:48,818 INFO [RS-EventLoopGroup-12-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:44180, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-11 14:20:48,819 INFO [Listener at localhost/45445] client.HBaseAdmin$15(890): Started disable of testTruncateWithFailoverAtStep4 2023-07-11 14:20:48,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45893] master.HMaster$11(2418): Client=jenkins//172.31.2.10 disable testTruncateWithFailoverAtStep4 2023-07-11 14:20:48,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45893] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-11 14:20:48,823 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085248823"}]},"ts":"1689085248823"} 2023-07-11 14:20:48,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45893] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-11 14:20:48,824 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=DISABLING in hbase:meta 2023-07-11 14:20:48,826 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testTruncateWithFailoverAtStep4 to state=DISABLING 2023-07-11 14:20:48,826 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=15ef30302259990af4c93ea5956e8be7, UNASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=deb98f3ca0218078c17361c806c6f249, UNASSIGN}, {pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=c887ddcecdd2e29691c3631bd0b10745, UNASSIGN}, {pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=6c361c95cb323d45992c0bfbbe2c6f34, UNASSIGN}] 2023-07-11 14:20:48,827 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=6c361c95cb323d45992c0bfbbe2c6f34, UNASSIGN 2023-07-11 14:20:48,828 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=6c361c95cb323d45992c0bfbbe2c6f34, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:48,828 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085248828"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085248828"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085248828"}]},"ts":"1689085248828"} 2023-07-11 14:20:48,830 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; CloseRegionProcedure 6c361c95cb323d45992c0bfbbe2c6f34, server=jenkins-hbase9.apache.org,33099,1689085243944}] 2023-07-11 14:20:48,831 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=c887ddcecdd2e29691c3631bd0b10745, UNASSIGN 2023-07-11 14:20:48,832 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=c887ddcecdd2e29691c3631bd0b10745, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:48,832 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085248832"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085248832"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085248832"}]},"ts":"1689085248832"} 2023-07-11 14:20:48,833 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=21, state=RUNNABLE; CloseRegionProcedure c887ddcecdd2e29691c3631bd0b10745, server=jenkins-hbase9.apache.org,33099,1689085243944}] 2023-07-11 14:20:48,835 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=deb98f3ca0218078c17361c806c6f249, UNASSIGN 2023-07-11 14:20:48,836 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=deb98f3ca0218078c17361c806c6f249, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:48,836 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085248836"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085248836"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085248836"}]},"ts":"1689085248836"} 2023-07-11 14:20:48,837 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=20, state=RUNNABLE; CloseRegionProcedure deb98f3ca0218078c17361c806c6f249, server=jenkins-hbase9.apache.org,33099,1689085243944}] 2023-07-11 14:20:48,840 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=15ef30302259990af4c93ea5956e8be7, UNASSIGN 2023-07-11 14:20:48,840 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=15ef30302259990af4c93ea5956e8be7, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:48,840 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085248840"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085248840"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085248840"}]},"ts":"1689085248840"} 2023-07-11 14:20:48,842 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=19, state=RUNNABLE; CloseRegionProcedure 15ef30302259990af4c93ea5956e8be7, server=jenkins-hbase9.apache.org,33099,1689085243944}] 2023-07-11 14:20:48,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45893] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-11 14:20:48,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(111): Close 15ef30302259990af4c93ea5956e8be7 2023-07-11 14:20:48,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 15ef30302259990af4c93ea5956e8be7, disabling compactions & flushes 2023-07-11 14:20:48,983 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7. 2023-07-11 14:20:48,983 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7. 2023-07-11 14:20:48,983 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7. after waiting 0 ms 2023-07-11 14:20:48,983 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7. 2023-07-11 14:20:48,983 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 15ef30302259990af4c93ea5956e8be7 2/2 column families, dataSize=7.98 KB heapSize=17.66 KB 2023-07-11 14:20:49,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45893] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-11 14:20:49,392 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.99 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/.tmp/f1/7d23079da6644a099c5a3b54f8493206 2023-07-11 14:20:49,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45893] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-11 14:20:49,806 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.99 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/.tmp/f2/e4670e9da6e145df8d320c88cb96108a 2023-07-11 14:20:49,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/.tmp/f1/7d23079da6644a099c5a3b54f8493206 as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/f1/7d23079da6644a099c5a3b54f8493206 2023-07-11 14:20:49,817 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/f1/7d23079da6644a099c5a3b54f8493206, entries=61, sequenceid=5, filesize=9.0 K 2023-07-11 14:20:49,818 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/.tmp/f2/e4670e9da6e145df8d320c88cb96108a as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/f2/e4670e9da6e145df8d320c88cb96108a 2023-07-11 14:20:49,823 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/f2/e4670e9da6e145df8d320c88cb96108a, entries=61, sequenceid=5, filesize=9.0 K 2023-07-11 14:20:49,824 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~7.98 KB/8174, heapSize ~17.63 KB/18048, currentSize=0 B/0 for 15ef30302259990af4c93ea5956e8be7 in 841ms, sequenceid=5, compaction requested=false 2023-07-11 14:20:49,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testTruncateWithFailoverAtStep4' 2023-07-11 14:20:49,829 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-11 14:20:49,830 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7. 2023-07-11 14:20:49,830 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 15ef30302259990af4c93ea5956e8be7: 2023-07-11 14:20:49,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(149): Closed 15ef30302259990af4c93ea5956e8be7 2023-07-11 14:20:49,832 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(111): Close c887ddcecdd2e29691c3631bd0b10745 2023-07-11 14:20:49,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing c887ddcecdd2e29691c3631bd0b10745, disabling compactions & flushes 2023-07-11 14:20:49,833 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745. 2023-07-11 14:20:49,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745. 2023-07-11 14:20:49,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745. after waiting 0 ms 2023-07-11 14:20:49,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745. 2023-07-11 14:20:49,833 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=15ef30302259990af4c93ea5956e8be7, regionState=CLOSED 2023-07-11 14:20:49,833 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing c887ddcecdd2e29691c3631bd0b10745 2/2 column families, dataSize=532 B heapSize=1.63 KB 2023-07-11 14:20:49,833 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085249833"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085249833"}]},"ts":"1689085249833"} 2023-07-11 14:20:49,836 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=19 2023-07-11 14:20:49,836 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=19, state=SUCCESS; CloseRegionProcedure 15ef30302259990af4c93ea5956e8be7, server=jenkins-hbase9.apache.org,33099,1689085243944 in 993 msec 2023-07-11 14:20:49,837 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=15ef30302259990af4c93ea5956e8be7, UNASSIGN in 1.0110 sec 2023-07-11 14:20:49,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45893] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-11 14:20:50,248 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=266 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/.tmp/f1/450705d87d5d40c38c120a4ff42c2e0d 2023-07-11 14:20:50,660 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=266 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/.tmp/f2/6e56630b0429411b9d9f3c12e8508f78 2023-07-11 14:20:50,666 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/.tmp/f1/450705d87d5d40c38c120a4ff42c2e0d as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/f1/450705d87d5d40c38c120a4ff42c2e0d 2023-07-11 14:20:50,671 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/f1/450705d87d5d40c38c120a4ff42c2e0d, entries=4, sequenceid=5, filesize=5.1 K 2023-07-11 14:20:50,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/.tmp/f2/6e56630b0429411b9d9f3c12e8508f78 as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/f2/6e56630b0429411b9d9f3c12e8508f78 2023-07-11 14:20:50,678 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/f2/6e56630b0429411b9d9f3c12e8508f78, entries=4, sequenceid=5, filesize=5.1 K 2023-07-11 14:20:50,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~532 B/532, heapSize ~1.59 KB/1632, currentSize=0 B/0 for c887ddcecdd2e29691c3631bd0b10745 in 846ms, sequenceid=5, compaction requested=false 2023-07-11 14:20:50,686 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-11 14:20:50,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745. 2023-07-11 14:20:50,687 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for c887ddcecdd2e29691c3631bd0b10745: 2023-07-11 14:20:50,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(149): Closed c887ddcecdd2e29691c3631bd0b10745 2023-07-11 14:20:50,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(111): Close deb98f3ca0218078c17361c806c6f249 2023-07-11 14:20:50,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing deb98f3ca0218078c17361c806c6f249, disabling compactions & flushes 2023-07-11 14:20:50,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249. 2023-07-11 14:20:50,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249. 2023-07-11 14:20:50,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249. after waiting 0 ms 2023-07-11 14:20:50,689 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=c887ddcecdd2e29691c3631bd0b10745, regionState=CLOSED 2023-07-11 14:20:50,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249. 2023-07-11 14:20:50,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing deb98f3ca0218078c17361c806c6f249 2/2 column families, dataSize=1.57 KB heapSize=3.88 KB 2023-07-11 14:20:50,689 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085250689"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085250689"}]},"ts":"1689085250689"} 2023-07-11 14:20:50,693 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=21 2023-07-11 14:20:50,693 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=21, state=SUCCESS; CloseRegionProcedure c887ddcecdd2e29691c3631bd0b10745, server=jenkins-hbase9.apache.org,33099,1689085243944 in 1.8580 sec 2023-07-11 14:20:50,694 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=c887ddcecdd2e29691c3631bd0b10745, UNASSIGN in 1.8670 sec 2023-07-11 14:20:50,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45893] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-11 14:20:51,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=802 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/.tmp/f1/05737c021038458a95218f3be231602e 2023-07-11 14:20:51,387 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-11 14:20:51,388 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-11 14:20:51,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=802 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/.tmp/f2/9d0ecd6e1b8140f085c7d0389aecb66c 2023-07-11 14:20:51,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/.tmp/f1/05737c021038458a95218f3be231602e as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/f1/05737c021038458a95218f3be231602e 2023-07-11 14:20:51,528 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/f1/05737c021038458a95218f3be231602e, entries=12, sequenceid=5, filesize=5.6 K 2023-07-11 14:20:51,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/.tmp/f2/9d0ecd6e1b8140f085c7d0389aecb66c as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/f2/9d0ecd6e1b8140f085c7d0389aecb66c 2023-07-11 14:20:51,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/f2/9d0ecd6e1b8140f085c7d0389aecb66c, entries=12, sequenceid=5, filesize=5.6 K 2023-07-11 14:20:51,535 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.57 KB/1604, heapSize ~3.84 KB/3936, currentSize=0 B/0 for deb98f3ca0218078c17361c806c6f249 in 845ms, sequenceid=5, compaction requested=false 2023-07-11 14:20:51,540 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-11 14:20:51,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249. 2023-07-11 14:20:51,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for deb98f3ca0218078c17361c806c6f249: 2023-07-11 14:20:51,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(149): Closed deb98f3ca0218078c17361c806c6f249 2023-07-11 14:20:51,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(111): Close 6c361c95cb323d45992c0bfbbe2c6f34 2023-07-11 14:20:51,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 6c361c95cb323d45992c0bfbbe2c6f34, disabling compactions & flushes 2023-07-11 14:20:51,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34. 2023-07-11 14:20:51,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34. 2023-07-11 14:20:51,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34. after waiting 0 ms 2023-07-11 14:20:51,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34. 2023-07-11 14:20:51,542 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=deb98f3ca0218078c17361c806c6f249, regionState=CLOSED 2023-07-11 14:20:51,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 6c361c95cb323d45992c0bfbbe2c6f34 2/2 column families, dataSize=3.01 KB heapSize=6.97 KB 2023-07-11 14:20:51,543 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085251542"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085251542"}]},"ts":"1689085251542"} 2023-07-11 14:20:51,545 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=20 2023-07-11 14:20:51,545 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=20, state=SUCCESS; CloseRegionProcedure deb98f3ca0218078c17361c806c6f249, server=jenkins-hbase9.apache.org,33099,1689085243944 in 2.7070 sec 2023-07-11 14:20:51,546 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=deb98f3ca0218078c17361c806c6f249, UNASSIGN in 2.7200 sec 2023-07-11 14:20:51,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.50 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/.tmp/f1/b4fda92b8a7447f6900429c3124f1140 2023-07-11 14:20:52,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.50 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/.tmp/f2/75da1c5e9baf4cc2b3e9cbbaf6013fca 2023-07-11 14:20:52,369 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/.tmp/f1/b4fda92b8a7447f6900429c3124f1140 as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/f1/b4fda92b8a7447f6900429c3124f1140 2023-07-11 14:20:52,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/f1/b4fda92b8a7447f6900429c3124f1140, entries=23, sequenceid=5, filesize=6.4 K 2023-07-11 14:20:52,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/.tmp/f2/75da1c5e9baf4cc2b3e9cbbaf6013fca as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/f2/75da1c5e9baf4cc2b3e9cbbaf6013fca 2023-07-11 14:20:52,379 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/f2/75da1c5e9baf4cc2b3e9cbbaf6013fca, entries=23, sequenceid=5, filesize=6.4 K 2023-07-11 14:20:52,380 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.01 KB/3078, heapSize ~6.94 KB/7104, currentSize=0 B/0 for 6c361c95cb323d45992c0bfbbe2c6f34 in 837ms, sequenceid=5, compaction requested=false 2023-07-11 14:20:52,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-11 14:20:52,385 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34. 2023-07-11 14:20:52,386 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 6c361c95cb323d45992c0bfbbe2c6f34: 2023-07-11 14:20:52,387 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(149): Closed 6c361c95cb323d45992c0bfbbe2c6f34 2023-07-11 14:20:52,387 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=6c361c95cb323d45992c0bfbbe2c6f34, regionState=CLOSED 2023-07-11 14:20:52,387 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085252387"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085252387"}]},"ts":"1689085252387"} 2023-07-11 14:20:52,390 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-11 14:20:52,390 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; CloseRegionProcedure 6c361c95cb323d45992c0bfbbe2c6f34, server=jenkins-hbase9.apache.org,33099,1689085243944 in 3.5590 sec 2023-07-11 14:20:52,392 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=18 2023-07-11 14:20:52,392 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=6c361c95cb323d45992c0bfbbe2c6f34, UNASSIGN in 3.5650 sec 2023-07-11 14:20:52,392 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085252392"}]},"ts":"1689085252392"} 2023-07-11 14:20:52,394 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=DISABLED in hbase:meta 2023-07-11 14:20:52,395 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testTruncateWithFailoverAtStep4 to state=DISABLED 2023-07-11 14:20:52,397 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 in 3.5770 sec 2023-07-11 14:20:52,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45893] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-11 14:20:52,932 INFO [Listener at localhost/45445] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testTruncateWithFailoverAtStep4, procId: 18 completed 2023-07-11 14:20:52,932 WARN [Listener at localhost/45445] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-07-11 14:20:52,938 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:TRUNCATE_TABLE_PRE_OPERATION; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:52,940 DEBUG [PEWorker-1] procedure.TruncateTableProcedure(87): waiting for 'testTruncateWithFailoverAtStep4' regions in transition 2023-07-11 14:20:52,940 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:52,940 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:52,940 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:52,940 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:53,146 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-11 14:20:53,188 INFO [Listener at localhost/45445] procedure.TestMasterFailoverWithProcedures(306): Restart 0 exec state: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:53,189 INFO [Listener at localhost/45445] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:20:53,189 INFO [Listener at localhost/45445] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:53,189 INFO [Listener at localhost/45445] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:53,191 ERROR [Listener at localhost/45445] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:53,191 INFO [Listener at localhost/45445] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:20:53,191 INFO [Listener at localhost/45445] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:53,191 INFO [Listener at localhost/45445] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:53,191 INFO [Listener at localhost/45445] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:53,192 INFO [Listener at localhost/45445] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:53,193 DEBUG [Listener at localhost/45445] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure) stack ids=[] 2023-07-11 14:20:53,193 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:53,194 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:53,194 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:53,194 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:53,194 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-11 14:20:53,195 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-11 14:20:53,195 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:TRUNCATE_TABLE_PRE_OPERATION; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:53,195 DEBUG [Listener at localhost/45445] procedure2.Procedure(966): pid=27, state=RUNNABLE:TRUNCATE_TABLE_PRE_OPERATION; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:53,195 INFO [Listener at localhost/45445] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-11 14:20:53,196 DEBUG [Listener at localhost/45445] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:53,196 DEBUG [Listener at localhost/45445] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,33099,1689085243944, table=hbase:meta, region=1588230740 2023-07-11 14:20:53,197 INFO [Listener at localhost/45445] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085283197 2023-07-11 14:20:53,197 DEBUG [Listener at localhost/45445] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:53,198 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:TRUNCATE_TABLE_PRE_OPERATION; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:53,203 DEBUG [Listener at localhost/45445] assignment.RegionStateStore(147): Load hbase:meta entry region=d0027ffc5e8dfd7f30f021f0fdb84921, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,33099,1689085243944, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944, openSeqNum=2 2023-07-11 14:20:53,203 DEBUG [Listener at localhost/45445] assignment.RegionStateStore(147): Load hbase:meta entry region=15ef30302259990af4c93ea5956e8be7, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,33099,1689085243944, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944, openSeqNum=2 2023-07-11 14:20:53,203 DEBUG [Listener at localhost/45445] assignment.RegionStateStore(147): Load hbase:meta entry region=deb98f3ca0218078c17361c806c6f249, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,33099,1689085243944, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944, openSeqNum=2 2023-07-11 14:20:53,203 DEBUG [Listener at localhost/45445] assignment.RegionStateStore(147): Load hbase:meta entry region=c887ddcecdd2e29691c3631bd0b10745, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,33099,1689085243944, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944, openSeqNum=2 2023-07-11 14:20:53,203 DEBUG [Listener at localhost/45445] assignment.RegionStateStore(147): Load hbase:meta entry region=6c361c95cb323d45992c0bfbbe2c6f34, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,33099,1689085243944, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944, openSeqNum=2 2023-07-11 14:20:53,203 INFO [Listener at localhost/45445] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:53,203 INFO [Listener at localhost/45445] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085313203 2023-07-11 14:20:53,204 INFO [Listener at localhost/45445] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085373204 2023-07-11 14:20:53,204 INFO [Listener at localhost/45445] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-07-11 14:20:53,204 DEBUG [PEWorker-1] procedure.TruncateTableProcedure(87): waiting for 'testTruncateWithFailoverAtStep4' regions in transition 2023-07-11 14:20:53,204 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:20:53,213 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7 2023-07-11 14:20:53,213 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745 2023-07-11 14:20:53,213 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34 2023-07-11 14:20:53,213 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249 2023-07-11 14:20:53,216 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/f1, FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/f2, FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/recovered.edits] 2023-07-11 14:20:53,216 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/f1, FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/f2, FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/recovered.edits] 2023-07-11 14:20:53,216 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/f1, FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/f2, FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/recovered.edits] 2023-07-11 14:20:53,216 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/f1, FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/f2, FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/recovered.edits] 2023-07-11 14:20:53,222 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/f1/7d23079da6644a099c5a3b54f8493206 to hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/archive/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/f1/7d23079da6644a099c5a3b54f8493206 2023-07-11 14:20:53,223 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/f1/450705d87d5d40c38c120a4ff42c2e0d to hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/archive/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/f1/450705d87d5d40c38c120a4ff42c2e0d 2023-07-11 14:20:53,223 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/f1/05737c021038458a95218f3be231602e to hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/archive/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/f1/05737c021038458a95218f3be231602e 2023-07-11 14:20:53,223 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/f1/b4fda92b8a7447f6900429c3124f1140 to hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/archive/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/f1/b4fda92b8a7447f6900429c3124f1140 2023-07-11 14:20:53,229 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/f2/e4670e9da6e145df8d320c88cb96108a to hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/archive/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/f2/e4670e9da6e145df8d320c88cb96108a 2023-07-11 14:20:53,230 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/f2/9d0ecd6e1b8140f085c7d0389aecb66c to hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/archive/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/f2/9d0ecd6e1b8140f085c7d0389aecb66c 2023-07-11 14:20:53,230 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/f2/6e56630b0429411b9d9f3c12e8508f78 to hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/archive/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/f2/6e56630b0429411b9d9f3c12e8508f78 2023-07-11 14:20:53,230 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/f2/75da1c5e9baf4cc2b3e9cbbaf6013fca to hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/archive/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/f2/75da1c5e9baf4cc2b3e9cbbaf6013fca 2023-07-11 14:20:53,235 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/recovered.edits/8.seqid to hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/archive/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7/recovered.edits/8.seqid 2023-07-11 14:20:53,235 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/recovered.edits/8.seqid to hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/archive/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745/recovered.edits/8.seqid 2023-07-11 14:20:53,235 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/recovered.edits/8.seqid to hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/archive/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249/recovered.edits/8.seqid 2023-07-11 14:20:53,236 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/recovered.edits/8.seqid to hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/archive/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34/recovered.edits/8.seqid 2023-07-11 14:20:53,237 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7 2023-07-11 14:20:53,237 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745 2023-07-11 14:20:53,238 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34 2023-07-11 14:20:53,238 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249 2023-07-11 14:20:53,238 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTruncateWithFailoverAtStep4 regions 2023-07-11 14:20:53,240 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:53,240 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_REMOVE_FROM_META, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:53,240 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:53,240 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_REMOVE_FROM_META, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:53,454 INFO [Listener at localhost/45445] procedure.TestMasterFailoverWithProcedures(306): Restart 1 exec state: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:53,454 INFO [Listener at localhost/45445] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:20:53,454 INFO [Listener at localhost/45445] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:53,454 INFO [Listener at localhost/45445] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:53,456 ERROR [Listener at localhost/45445] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:53,456 INFO [Listener at localhost/45445] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:20:53,456 INFO [Listener at localhost/45445] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:53,456 INFO [Listener at localhost/45445] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:53,456 INFO [Listener at localhost/45445] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:53,457 INFO [Listener at localhost/45445] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:53,458 DEBUG [Listener at localhost/45445] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure) stack ids=[0] 2023-07-11 14:20:53,458 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:53,458 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:53,458 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:53,459 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:53,459 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-11 14:20:53,459 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-11 14:20:53,459 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:53,460 DEBUG [Listener at localhost/45445] procedure2.Procedure(966): pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:53,460 INFO [Listener at localhost/45445] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-11 14:20:53,460 DEBUG [Listener at localhost/45445] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:53,461 DEBUG [Listener at localhost/45445] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,33099,1689085243944, table=hbase:meta, region=1588230740 2023-07-11 14:20:53,461 INFO [Listener at localhost/45445] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085283461 2023-07-11 14:20:53,462 DEBUG [Listener at localhost/45445] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:53,462 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:53,464 DEBUG [Listener at localhost/45445] assignment.RegionStateStore(147): Load hbase:meta entry region=d0027ffc5e8dfd7f30f021f0fdb84921, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,33099,1689085243944, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944, openSeqNum=2 2023-07-11 14:20:53,465 DEBUG [Listener at localhost/45445] assignment.RegionStateStore(147): Load hbase:meta entry region=15ef30302259990af4c93ea5956e8be7, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,33099,1689085243944, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944, openSeqNum=2 2023-07-11 14:20:53,465 DEBUG [Listener at localhost/45445] assignment.RegionStateStore(147): Load hbase:meta entry region=deb98f3ca0218078c17361c806c6f249, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,33099,1689085243944, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944, openSeqNum=2 2023-07-11 14:20:53,465 DEBUG [Listener at localhost/45445] assignment.RegionStateStore(147): Load hbase:meta entry region=c887ddcecdd2e29691c3631bd0b10745, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,33099,1689085243944, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944, openSeqNum=2 2023-07-11 14:20:53,465 DEBUG [Listener at localhost/45445] assignment.RegionStateStore(147): Load hbase:meta entry region=6c361c95cb323d45992c0bfbbe2c6f34, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,33099,1689085243944, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944, openSeqNum=2 2023-07-11 14:20:53,465 INFO [Listener at localhost/45445] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:53,465 INFO [Listener at localhost/45445] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085313465 2023-07-11 14:20:53,465 INFO [Listener at localhost/45445] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085373465 2023-07-11 14:20:53,465 INFO [Listener at localhost/45445] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-11 14:20:53,470 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7 2023-07-11 14:20:53,470 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34 2023-07-11 14:20:53,470 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745 2023-07-11 14:20:53,470 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249 2023-07-11 14:20:53,470 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7 empty. 2023-07-11 14:20:53,470 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34 empty. 2023-07-11 14:20:53,470 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745 empty. 2023-07-11 14:20:53,470 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249 empty. 2023-07-11 14:20:53,471 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/15ef30302259990af4c93ea5956e8be7 2023-07-11 14:20:53,471 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/c887ddcecdd2e29691c3631bd0b10745 2023-07-11 14:20:53,471 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/6c361c95cb323d45992c0bfbbe2c6f34 2023-07-11 14:20:53,471 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/deb98f3ca0218078c17361c806c6f249 2023-07-11 14:20:53,471 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTruncateWithFailoverAtStep4 regions 2023-07-11 14:20:53,473 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:20:53,477 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 4 rows of testTruncateWithFailoverAtStep4 from hbase:meta 2023-07-11 14:20:53,479 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testTruncateWithFailoverAtStep4' descriptor. 2023-07-11 14:20:53,479 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testTruncateWithFailoverAtStep4' from region states. 2023-07-11 14:20:53,479 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689085253479"}]},"ts":"9223372036854775807"} 2023-07-11 14:20:53,479 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689085253479"}]},"ts":"9223372036854775807"} 2023-07-11 14:20:53,479 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689085253479"}]},"ts":"9223372036854775807"} 2023-07-11 14:20:53,479 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689085253479"}]},"ts":"9223372036854775807"} 2023-07-11 14:20:53,481 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 4 regions from META 2023-07-11 14:20:53,481 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 15ef30302259990af4c93ea5956e8be7, NAME => 'testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7.', STARTKEY => '', ENDKEY => 'a'}, {ENCODED => deb98f3ca0218078c17361c806c6f249, NAME => 'testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249.', STARTKEY => 'a', ENDKEY => 'b'}, {ENCODED => c887ddcecdd2e29691c3631bd0b10745, NAME => 'testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745.', STARTKEY => 'b', ENDKEY => 'c'}, {ENCODED => 6c361c95cb323d45992c0bfbbe2c6f34, NAME => 'testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34.', STARTKEY => 'c', ENDKEY => ''}] 2023-07-11 14:20:53,481 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testTruncateWithFailoverAtStep4' as deleted. 2023-07-11 14:20:53,482 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689085253481"}]},"ts":"9223372036854775807"} 2023-07-11 14:20:53,483 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testTruncateWithFailoverAtStep4 state from META 2023-07-11 14:20:53,485 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:53,485 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CREATE_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:53,485 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:53,485 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CREATE_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:53,716 INFO [Listener at localhost/45445] procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:53,716 INFO [Listener at localhost/45445] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:20:53,716 INFO [Listener at localhost/45445] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:53,716 INFO [Listener at localhost/45445] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:53,716 ERROR [Listener at localhost/45445] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:53,716 INFO [Listener at localhost/45445] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:20:53,716 INFO [Listener at localhost/45445] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:53,716 INFO [Listener at localhost/45445] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:53,716 INFO [Listener at localhost/45445] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:53,717 INFO [Listener at localhost/45445] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:53,718 DEBUG [Listener at localhost/45445] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure) stack ids=[0, 1] 2023-07-11 14:20:53,718 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:53,719 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:53,719 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:53,719 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:53,719 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-11 14:20:53,720 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-11 14:20:53,720 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:TRUNCATE_TABLE_REMOVE_FROM_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:53,720 DEBUG [Listener at localhost/45445] procedure2.Procedure(966): pid=27, state=RUNNABLE:TRUNCATE_TABLE_REMOVE_FROM_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:53,720 INFO [Listener at localhost/45445] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-11 14:20:53,721 DEBUG [Listener at localhost/45445] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:53,721 DEBUG [Listener at localhost/45445] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,33099,1689085243944, table=hbase:meta, region=1588230740 2023-07-11 14:20:53,722 INFO [Listener at localhost/45445] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085283722 2023-07-11 14:20:53,722 DEBUG [Listener at localhost/45445] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:53,723 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:TRUNCATE_TABLE_REMOVE_FROM_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:53,725 DEBUG [Listener at localhost/45445] assignment.RegionStateStore(147): Load hbase:meta entry region=d0027ffc5e8dfd7f30f021f0fdb84921, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,33099,1689085243944, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944, openSeqNum=2 2023-07-11 14:20:53,725 INFO [Listener at localhost/45445] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:53,725 INFO [Listener at localhost/45445] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085313725 2023-07-11 14:20:53,725 INFO [Listener at localhost/45445] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085373725 2023-07-11 14:20:53,725 INFO [Listener at localhost/45445] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-11 14:20:53,727 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testTruncateWithFailoverAtStep4' descriptor. 2023-07-11 14:20:53,727 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testTruncateWithFailoverAtStep4' from region states. 2023-07-11 14:20:53,728 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 0 regions from META 2023-07-11 14:20:53,728 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [] 2023-07-11 14:20:53,728 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testTruncateWithFailoverAtStep4' as deleted. 2023-07-11 14:20:53,728 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689085253728"}]},"ts":"9223372036854775807"} 2023-07-11 14:20:53,729 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testTruncateWithFailoverAtStep4 state from META 2023-07-11 14:20:53,731 DEBUG [PEWorker-1] zookeeper.RecoverableZooKeeper(172): Node /hbase/table/testTruncateWithFailoverAtStep4 already deleted, retry=false 2023-07-11 14:20:53,731 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:20:53,737 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65 2023-07-11 14:20:53,737 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:20:53,737 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1 2023-07-11 14:20:53,737 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:20:53,737 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65 empty. 2023-07-11 14:20:53,737 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994 empty. 2023-07-11 14:20:53,737 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d empty. 2023-07-11 14:20:53,737 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1 empty. 2023-07-11 14:20:53,738 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65 2023-07-11 14:20:53,738 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:20:53,738 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:20:53,738 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1 2023-07-11 14:20:53,738 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTruncateWithFailoverAtStep4 regions 2023-07-11 14:20:54,147 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:54,149 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(7675): creating {ENCODED => aa3553f00fe04c2619619d4fa27dd51d, NAME => 'testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d.', STARTKEY => 'b', ENDKEY => 'c'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp 2023-07-11 14:20:54,149 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(7675): creating {ENCODED => a160933024ed654b6b55eac434a2d2d1, NAME => 'testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1.', STARTKEY => 'a', ENDKEY => 'b'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp 2023-07-11 14:20:54,149 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9273ec20102dfbe67a6417ad2c513b65, NAME => 'testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65.', STARTKEY => '', ENDKEY => 'a'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp 2023-07-11 14:20:54,519 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-11 14:20:54,519 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-11 14:20:54,568 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:54,568 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:54,568 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1604): Closing aa3553f00fe04c2619619d4fa27dd51d, disabling compactions & flushes 2023-07-11 14:20:54,568 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1604): Closing a160933024ed654b6b55eac434a2d2d1, disabling compactions & flushes 2023-07-11 14:20:54,568 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:54,568 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. 2023-07-11 14:20:54,568 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1604): Closing 9273ec20102dfbe67a6417ad2c513b65, disabling compactions & flushes 2023-07-11 14:20:54,568 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. 2023-07-11 14:20:54,568 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. 2023-07-11 14:20:54,568 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. after waiting 0 ms 2023-07-11 14:20:54,568 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. 2023-07-11 14:20:54,568 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. 2023-07-11 14:20:54,568 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. 2023-07-11 14:20:54,568 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. 2023-07-11 14:20:54,568 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1558): Region close journal for a160933024ed654b6b55eac434a2d2d1: 2023-07-11 14:20:54,568 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. 2023-07-11 14:20:54,568 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. after waiting 0 ms 2023-07-11 14:20:54,569 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. 2023-07-11 14:20:54,569 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. 2023-07-11 14:20:54,569 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1558): Region close journal for 9273ec20102dfbe67a6417ad2c513b65: 2023-07-11 14:20:54,569 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. after waiting 0 ms 2023-07-11 14:20:54,569 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(7675): creating {ENCODED => ce18e850fe81c0f0bf4b2c1436f84994, NAME => 'testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994.', STARTKEY => 'c', ENDKEY => ''}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp 2023-07-11 14:20:54,569 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. 2023-07-11 14:20:54,569 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. 2023-07-11 14:20:54,569 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1558): Region close journal for aa3553f00fe04c2619619d4fa27dd51d: 2023-07-11 14:20:54,977 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:54,977 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1604): Closing ce18e850fe81c0f0bf4b2c1436f84994, disabling compactions & flushes 2023-07-11 14:20:54,977 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. 2023-07-11 14:20:54,977 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. 2023-07-11 14:20:54,977 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. after waiting 0 ms 2023-07-11 14:20:54,977 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. 2023-07-11 14:20:54,977 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. 2023-07-11 14:20:54,977 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1558): Region close journal for ce18e850fe81c0f0bf4b2c1436f84994: 2023-07-11 14:20:54,978 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:54,979 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_ADD_TO_META, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:54,979 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:54,979 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_ADD_TO_META, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:55,226 INFO [Listener at localhost/45445] procedure.TestMasterFailoverWithProcedures(306): Restart 3 exec state: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:55,226 INFO [Listener at localhost/45445] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:20:55,226 INFO [Listener at localhost/45445] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:20:55,226 INFO [Listener at localhost/45445] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:55,226 ERROR [Listener at localhost/45445] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:55,226 INFO [Listener at localhost/45445] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:20:55,226 INFO [Listener at localhost/45445] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:55,226 INFO [Listener at localhost/45445] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:55,226 INFO [Listener at localhost/45445] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:55,227 INFO [Listener at localhost/45445] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:55,228 DEBUG [Listener at localhost/45445] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure) stack ids=[0, 1, 2] 2023-07-11 14:20:55,228 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:55,228 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:55,229 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:55,229 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:55,229 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-11 14:20:55,229 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-11 14:20:55,230 DEBUG [Listener at localhost/45445] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:TRUNCATE_TABLE_CREATE_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:55,230 DEBUG [Listener at localhost/45445] procedure2.Procedure(966): pid=27, state=RUNNABLE:TRUNCATE_TABLE_CREATE_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:55,230 INFO [Listener at localhost/45445] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-11 14:20:55,230 DEBUG [Listener at localhost/45445] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:55,231 DEBUG [Listener at localhost/45445] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,33099,1689085243944, table=hbase:meta, region=1588230740 2023-07-11 14:20:55,231 INFO [Listener at localhost/45445] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085285231 2023-07-11 14:20:55,232 DEBUG [Listener at localhost/45445] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:55,232 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:TRUNCATE_TABLE_CREATE_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:55,234 DEBUG [Listener at localhost/45445] assignment.RegionStateStore(147): Load hbase:meta entry region=d0027ffc5e8dfd7f30f021f0fdb84921, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,33099,1689085243944, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944, openSeqNum=2 2023-07-11 14:20:55,234 INFO [Listener at localhost/45445] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:55,234 INFO [Listener at localhost/45445] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085315234 2023-07-11 14:20:55,234 INFO [Listener at localhost/45445] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085375234 2023-07-11 14:20:55,234 INFO [Listener at localhost/45445] assignment.AssignmentManager(1536): Joined the cluster in 2 msec 2023-07-11 14:20:55,239 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65 2023-07-11 14:20:55,239 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:20:55,239 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:20:55,239 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1 2023-07-11 14:20:55,240 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65 empty. 2023-07-11 14:20:55,240 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994 empty. 2023-07-11 14:20:55,241 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1 empty. 2023-07-11 14:20:55,241 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d empty. 2023-07-11 14:20:55,241 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65 2023-07-11 14:20:55,241 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1 2023-07-11 14:20:55,242 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:20:55,242 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:20:55,242 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTruncateWithFailoverAtStep4 regions 2023-07-11 14:20:55,651 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp/data/default/testTruncateWithFailoverAtStep4/.tabledesc/.tableinfo.0000000001 2023-07-11 14:20:55,652 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(7675): creating {ENCODED => aa3553f00fe04c2619619d4fa27dd51d, NAME => 'testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d.', STARTKEY => 'b', ENDKEY => 'c'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp 2023-07-11 14:20:55,653 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9273ec20102dfbe67a6417ad2c513b65, NAME => 'testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65.', STARTKEY => '', ENDKEY => 'a'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp 2023-07-11 14:20:55,653 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(7675): creating {ENCODED => a160933024ed654b6b55eac434a2d2d1, NAME => 'testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1.', STARTKEY => 'a', ENDKEY => 'b'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp 2023-07-11 14:20:56,059 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:56,059 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:56,060 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1604): Closing a160933024ed654b6b55eac434a2d2d1, disabling compactions & flushes 2023-07-11 14:20:56,060 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1604): Closing 9273ec20102dfbe67a6417ad2c513b65, disabling compactions & flushes 2023-07-11 14:20:56,060 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:56,060 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. 2023-07-11 14:20:56,060 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1604): Closing aa3553f00fe04c2619619d4fa27dd51d, disabling compactions & flushes 2023-07-11 14:20:56,060 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. 2023-07-11 14:20:56,060 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. 2023-07-11 14:20:56,060 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. after waiting 0 ms 2023-07-11 14:20:56,060 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. 2023-07-11 14:20:56,060 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. 2023-07-11 14:20:56,060 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. 2023-07-11 14:20:56,060 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. 2023-07-11 14:20:56,060 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1558): Region close journal for 9273ec20102dfbe67a6417ad2c513b65: 2023-07-11 14:20:56,060 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. 2023-07-11 14:20:56,060 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. after waiting 0 ms 2023-07-11 14:20:56,060 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. 2023-07-11 14:20:56,060 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. 2023-07-11 14:20:56,060 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1558): Region close journal for a160933024ed654b6b55eac434a2d2d1: 2023-07-11 14:20:56,061 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. after waiting 0 ms 2023-07-11 14:20:56,061 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. 2023-07-11 14:20:56,061 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(7675): creating {ENCODED => ce18e850fe81c0f0bf4b2c1436f84994, NAME => 'testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994.', STARTKEY => 'c', ENDKEY => ''}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.tmp 2023-07-11 14:20:56,061 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. 2023-07-11 14:20:56,061 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1558): Region close journal for aa3553f00fe04c2619619d4fa27dd51d: 2023-07-11 14:20:56,467 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:56,467 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1604): Closing ce18e850fe81c0f0bf4b2c1436f84994, disabling compactions & flushes 2023-07-11 14:20:56,467 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. 2023-07-11 14:20:56,467 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. 2023-07-11 14:20:56,467 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. after waiting 0 ms 2023-07-11 14:20:56,467 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. 2023-07-11 14:20:56,467 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. 2023-07-11 14:20:56,467 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1558): Region close journal for ce18e850fe81c0f0bf4b2c1436f84994: 2023-07-11 14:20:56,469 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:20:56,472 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085256472"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085256472"}]},"ts":"1689085256472"} 2023-07-11 14:20:56,472 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085256472"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085256472"}]},"ts":"1689085256472"} 2023-07-11 14:20:56,472 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085256472"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085256472"}]},"ts":"1689085256472"} 2023-07-11 14:20:56,472 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085256472"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085256472"}]},"ts":"1689085256472"} 2023-07-11 14:20:56,474 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 4 regions to meta. 2023-07-11 14:20:56,474 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:20:56,474 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_ASSIGN_REGIONS, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:56,474 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:56,474 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_ASSIGN_REGIONS, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:20:56,485 INFO [Listener at localhost/45445] procedure.TestMasterFailoverWithProcedures(313): Trigger master failover 2023-07-11 14:20:56,485 INFO [Listener at localhost/45445] hbase.MiniHBaseCluster(607): Aborting jenkins-hbase9.apache.org,45893,1689085243884 2023-07-11 14:20:56,485 ERROR [Listener at localhost/45445] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-11 14:20:56,485 ERROR [Listener at localhost/45445] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase9.apache.org,45893,1689085243884: Aborting for tests ***** java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:608) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:411) at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.masterFailover(MasterProcedureTestingUtility.java:135) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testRecoveryAndDoubleExecution(TestMasterFailoverWithProcedures.java:314) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testTruncateWithFailoverAtStep(TestMasterFailoverWithProcedures.java:201) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testTruncateWithFailover(TestMasterFailoverWithProcedures.java:175) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-11 14:20:56,485 INFO [Listener at localhost/45445] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,45893,1689085243884' ***** 2023-07-11 14:20:56,485 INFO [Listener at localhost/45445] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost/45445 2023-07-11 14:20:56,486 DEBUG [M:0;jenkins-hbase9:45893] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@680bd66b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-11 14:20:56,486 INFO [M:0;jenkins-hbase9:45893] regionserver.HRegionServer(1141): aborting server jenkins-hbase9.apache.org,45893,1689085243884 2023-07-11 14:20:56,486 DEBUG [M:0;jenkins-hbase9:45893] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44c19f59 to 127.0.0.1:50076 2023-07-11 14:20:56,486 DEBUG [M:0;jenkins-hbase9:45893] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:56,486 INFO [M:0;jenkins-hbase9:45893] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,45893,1689085243884; all regions closed. 2023-07-11 14:20:56,486 DEBUG [M:0;jenkins-hbase9:45893] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:56,486 DEBUG [M:0;jenkins-hbase9:45893] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-11 14:20:56,487 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-11 14:20:56,487 DEBUG [M:0;jenkins-hbase9:45893] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-11 14:20:56,487 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085245294] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085245294,5,FailOnTimeoutGroup] 2023-07-11 14:20:56,487 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085245290] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085245290,5,FailOnTimeoutGroup] 2023-07-11 14:20:56,487 INFO [M:0;jenkins-hbase9:45893] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-11 14:20:56,488 INFO [M:0;jenkins-hbase9:45893] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-11 14:20:56,488 INFO [M:0;jenkins-hbase9:45893] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-07-11 14:20:56,488 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): regionserver:33099-0x10154b433ed0002, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:56,488 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:56,488 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:56,488 DEBUG [M:0;jenkins-hbase9:45893] master.HMaster(1512): Stopping service threads 2023-07-11 14:20:56,488 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:56,488 INFO [M:0;jenkins-hbase9:45893] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-11 14:20:56,488 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:56,489 ERROR [M:0;jenkins-hbase9:45893] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:20:56,489 INFO [M:0;jenkins-hbase9:45893] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-11 14:20:56,489 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-11 14:20:56,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:56,490 DEBUG [M:0;jenkins-hbase9:45893] zookeeper.ZKUtil(398): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-11 14:20:56,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:56,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33099-0x10154b433ed0002, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:56,490 WARN [M:0;jenkins-hbase9:45893] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-11 14:20:56,490 INFO [M:0;jenkins-hbase9:45893] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:20:56,491 INFO [M:0;jenkins-hbase9:45893] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-11 14:20:56,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:20:56,491 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-11 14:20:56,492 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:56,492 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:56,492 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:56,493 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:56,492 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): regionserver:33099-0x10154b433ed0002, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:20:56,493 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:56,493 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,38655,1689085243913 from backup master directory 2023-07-11 14:20:56,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:20:56,494 DEBUG [M:0;jenkins-hbase9:45893] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 14:20:56,494 INFO [M:0;jenkins-hbase9:45893] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:56,494 DEBUG [M:0;jenkins-hbase9:45893] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:56,494 DEBUG [M:0;jenkins-hbase9:45893] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 14:20:56,494 DEBUG [M:0;jenkins-hbase9:45893] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:56,494 ERROR [M:0;jenkins-hbase9:45893] regionserver.HRegion(1825): Memstore data size is 92223 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:56,495 INFO [M:0;jenkins-hbase9:45893] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:20:56,495 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:56,495 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,38655,1689085243913 2023-07-11 14:20:56,495 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 14:20:56,495 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase9.apache.org,38655,1689085243913 2023-07-11 14:20:56,495 DEBUG [M:0;jenkins-hbase9:45893] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:56,495 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:20:56,495 INFO [M:0;jenkins-hbase9:45893] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-11 14:20:56,495 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 14:20:56,495 INFO [M:0;jenkins-hbase9:45893] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:45893 2023-07-11 14:20:56,497 DEBUG [M:0;jenkins-hbase9:45893] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,45893,1689085243884 already deleted, retry=false 2023-07-11 14:20:56,509 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:20:56,578 WARN [RS-EventLoopGroup-14-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:45893 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:45893 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-11 14:20:56,579 DEBUG [RS-EventLoopGroup-14-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:45893 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:45893 2023-07-11 14:20:56,680 DEBUG [RS:0;jenkins-hbase9:33099] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-11 14:20:56,681 INFO [RS-EventLoopGroup-13-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:50099, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=RegionServerStatusService 2023-07-11 14:20:56,682 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38655] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:56,690 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:56,690 INFO [M:0;jenkins-hbase9:45893] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,45893,1689085243884; zookeeper connection closed. 2023-07-11 14:20:56,690 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:45893-0x10154b433ed0000, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:20:56,784 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38655] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:56,885 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38655] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:56,925 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x43560ec4 to 127.0.0.1:50076 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:20:56,929 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@562d0466, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:20:56,929 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-11 14:20:56,930 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-11 14:20:56,930 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:20:56,935 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/WALs/jenkins-hbase9.apache.org,45893,1689085243884 to hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/WALs/jenkins-hbase9.apache.org,45893,1689085243884-dead as it is dead 2023-07-11 14:20:56,936 INFO [master/jenkins-hbase9:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/WALs/jenkins-hbase9.apache.org,45893,1689085243884-dead/jenkins-hbase9.apache.org%2C45893%2C1689085243884.1689085245214 2023-07-11 14:20:56,936 INFO [master/jenkins-hbase9:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/WALs/jenkins-hbase9.apache.org,45893,1689085243884-dead/jenkins-hbase9.apache.org%2C45893%2C1689085243884.1689085245214 after 0ms 2023-07-11 14:20:56,937 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/WALs/jenkins-hbase9.apache.org,45893,1689085243884-dead/jenkins-hbase9.apache.org%2C45893%2C1689085243884.1689085245214 to hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C45893%2C1689085243884.1689085245214 2023-07-11 14:20:56,937 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/WALs/jenkins-hbase9.apache.org,45893,1689085243884-dead 2023-07-11 14:20:56,937 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/WALs/jenkins-hbase9.apache.org,38655,1689085243913 2023-07-11 14:20:56,939 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C38655%2C1689085243913, suffix=, logDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/WALs/jenkins-hbase9.apache.org,38655,1689085243913, archiveDir=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/oldWALs, maxLogs=10 2023-07-11 14:20:56,952 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33177,DS-cd205a8b-0c62-4681-a092-17d405a77c99,DISK] 2023-07-11 14:20:56,954 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/WALs/jenkins-hbase9.apache.org,38655,1689085243913/jenkins-hbase9.apache.org%2C38655%2C1689085243913.1689085256939 2023-07-11 14:20:56,954 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33177,DS-cd205a8b-0c62-4681-a092-17d405a77c99,DISK]] 2023-07-11 14:20:56,954 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:20:56,954 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:56,954 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:56,954 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:56,955 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:56,956 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-11 14:20:56,956 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-11 14:20:56,957 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:56,957 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-11 14:20:56,957 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C45893%2C1689085243884.1689085245214 2023-07-11 14:20:56,964 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 232, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=194, path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C45893%2C1689085243884.1689085245214 2023-07-11 14:20:56,964 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=90.06 KB heapSize=107.49 KB 2023-07-11 14:20:56,987 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38655] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:57,089 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38655] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:57,191 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38655] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:57,292 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38655] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:57,372 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=90.06 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/62629206b72f43b8b6ee8844ae2448de 2023-07-11 14:20:57,377 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/62629206b72f43b8b6ee8844ae2448de as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/62629206b72f43b8b6ee8844ae2448de 2023-07-11 14:20:57,381 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/62629206b72f43b8b6ee8844ae2448de, entries=232, sequenceid=194, filesize=95.3 K 2023-07-11 14:20:57,382 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~90.06 KB/92223, heapSize ~107.48 KB/110056, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 418ms, sequenceid=194, compaction requested=false; wal=null 2023-07-11 14:20:57,382 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C45893%2C1689085243884.1689085245214 2023-07-11 14:20:57,385 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:20:57,387 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/194.seqid, newMaxSeqId=194, maxSeqId=1 2023-07-11 14:20:57,388 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=195; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72858310, jitterRate=0.08567342162132263}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 14:20:57,388 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:20:57,388 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-11 14:20:57,389 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:20:57,389 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:20:57,389 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:20:57,390 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:20:57,394 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38655] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:20:57,394 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure) stack ids=[0, 1, 2, 3] 2023-07-11 14:20:57,395 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:20:57,395 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:20:57,395 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:20:57,395 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:20:57,395 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-11 14:20:57,396 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-11 14:20:57,396 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:TRUNCATE_TABLE_ADD_TO_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:57,396 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.Procedure(966): pid=27, state=RUNNABLE:TRUNCATE_TABLE_ADD_TO_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:20:57,396 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-07-11 14:20:57,396 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-11 14:20:57,397 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:20:57,397 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,33099,1689085243944, table=hbase:meta, region=1588230740 2023-07-11 14:20:57,398 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 1 possibly 'live' servers, and 0 'splitting'. 2023-07-11 14:20:57,399 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:57,409 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-11 14:20:57,409 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-11 14:20:57,410 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-11 14:20:57,410 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-11 14:20:57,410 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-11 14:20:57,411 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-11 14:20:57,411 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-11 14:20:57,412 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-11 14:20:57,412 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase9.apache.org,38655,1689085243913, sessionid=0x10154b433ed0001, setting cluster-up flag (Was=true) 2023-07-11 14:20:57,414 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-11 14:20:57,415 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,38655,1689085243913 2023-07-11 14:20:57,417 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-11 14:20:57,418 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,38655,1689085243913 2023-07-11 14:20:57,418 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/.hbase-snapshot/.tmp 2023-07-11 14:20:57,419 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:57,419 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:57,419 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:57,419 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:20:57,419 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-07-11 14:20:57,419 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:57,419 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-11 14:20:57,419 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:20:57,419 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085287419 2023-07-11 14:20:57,420 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-11 14:20:57,420 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:TRUNCATE_TABLE_ADD_TO_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-11 14:20:57,420 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-11 14:20:57,421 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-11 14:20:57,421 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-11 14:20:57,421 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-11 14:20:57,421 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-11 14:20:57,421 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:57,421 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-11 14:20:57,421 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-11 14:20:57,421 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-11 14:20:57,421 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-11 14:20:57,422 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-11 14:20:57,422 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085257422,5,FailOnTimeoutGroup] 2023-07-11 14:20:57,422 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085257422,5,FailOnTimeoutGroup] 2023-07-11 14:20:57,422 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:57,422 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-11 14:20:57,422 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:57,422 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:57,422 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689085257422, completionTime=-1 2023-07-11 14:20:57,422 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-11 14:20:57,422 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:20:57,423 DEBUG [hconnection-0x21d4f7fe-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 14:20:57,424 INFO [RS-EventLoopGroup-14-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:48918, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 14:20:57,426 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=d0027ffc5e8dfd7f30f021f0fdb84921, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,33099,1689085243944, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944, openSeqNum=2 2023-07-11 14:20:57,426 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=9273ec20102dfbe67a6417ad2c513b65, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-11 14:20:57,426 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=a160933024ed654b6b55eac434a2d2d1, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-11 14:20:57,426 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=aa3553f00fe04c2619619d4fa27dd51d, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-11 14:20:57,426 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=ce18e850fe81c0f0bf4b2c1436f84994, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-11 14:20:57,427 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:20:57,427 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085317427 2023-07-11 14:20:57,427 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085377427 2023-07-11 14:20:57,427 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-07-11 14:20:57,432 WARN [master/jenkins-hbase9:0:becomeActiveMaster] master.TableStateManager(233): testTruncateWithFailoverAtStep4 has no table state in hbase:meta, assuming ENABLED 2023-07-11 14:20:57,432 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085257432"}]},"ts":"1689085257432"} 2023-07-11 14:20:57,434 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=ENABLED in hbase:meta 2023-07-11 14:20:57,441 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,38655,1689085243913-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:57,441 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,38655,1689085243913-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:57,441 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,38655,1689085243913-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:57,442 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:38655, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:57,442 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-11 14:20:57,442 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-11 14:20:57,444 DEBUG [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-11 14:20:57,452 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-11 14:20:57,456 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-11 14:20:57,456 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 0.961sec 2023-07-11 14:20:57,456 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-11 14:20:57,456 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-11 14:20:57,456 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-11 14:20:57,456 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,38655,1689085243913-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-11 14:20:57,456 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,38655,1689085243913-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-11 14:20:57,457 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085257457"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085257457"}]},"ts":"1689085257457"} 2023-07-11 14:20:57,457 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085257457"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085257457"}]},"ts":"1689085257457"} 2023-07-11 14:20:57,457 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085257457"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085257457"}]},"ts":"1689085257457"} 2023-07-11 14:20:57,457 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085257457"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085257457"}]},"ts":"1689085257457"} 2023-07-11 14:20:57,460 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 4 regions to meta. 2023-07-11 14:20:57,462 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085257462"}]},"ts":"1689085257462"} 2023-07-11 14:20:57,462 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-11 14:20:57,463 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=ENABLING in hbase:meta 2023-07-11 14:20:57,469 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=9273ec20102dfbe67a6417ad2c513b65, ASSIGN}, {pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=a160933024ed654b6b55eac434a2d2d1, ASSIGN}, {pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=aa3553f00fe04c2619619d4fa27dd51d, ASSIGN}, {pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=ce18e850fe81c0f0bf4b2c1436f84994, ASSIGN}] 2023-07-11 14:20:57,470 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=ce18e850fe81c0f0bf4b2c1436f84994, ASSIGN 2023-07-11 14:20:57,471 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=ce18e850fe81c0f0bf4b2c1436f84994, ASSIGN; state=CLOSED, location=jenkins-hbase9.apache.org,33099,1689085243944; forceNewPlan=false, retain=false 2023-07-11 14:20:57,471 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=aa3553f00fe04c2619619d4fa27dd51d, ASSIGN 2023-07-11 14:20:57,472 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=aa3553f00fe04c2619619d4fa27dd51d, ASSIGN; state=CLOSED, location=jenkins-hbase9.apache.org,33099,1689085243944; forceNewPlan=false, retain=false 2023-07-11 14:20:57,472 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=a160933024ed654b6b55eac434a2d2d1, ASSIGN 2023-07-11 14:20:57,472 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=a160933024ed654b6b55eac434a2d2d1, ASSIGN; state=CLOSED, location=jenkins-hbase9.apache.org,33099,1689085243944; forceNewPlan=false, retain=false 2023-07-11 14:20:57,473 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=9273ec20102dfbe67a6417ad2c513b65, ASSIGN 2023-07-11 14:20:57,473 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=9273ec20102dfbe67a6417ad2c513b65, ASSIGN; state=CLOSED, location=jenkins-hbase9.apache.org,33099,1689085243944; forceNewPlan=false, retain=false 2023-07-11 14:20:57,623 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=9273ec20102dfbe67a6417ad2c513b65, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:57,623 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085257622"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085257622"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085257622"}]},"ts":"1689085257622"} 2023-07-11 14:20:57,624 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=28, state=RUNNABLE; OpenRegionProcedure 9273ec20102dfbe67a6417ad2c513b65, server=jenkins-hbase9.apache.org,33099,1689085243944}] 2023-07-11 14:20:57,627 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=a160933024ed654b6b55eac434a2d2d1, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:57,627 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085257627"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085257627"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085257627"}]},"ts":"1689085257627"} 2023-07-11 14:20:57,629 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=29, state=RUNNABLE; OpenRegionProcedure a160933024ed654b6b55eac434a2d2d1, server=jenkins-hbase9.apache.org,33099,1689085243944}] 2023-07-11 14:20:57,631 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=aa3553f00fe04c2619619d4fa27dd51d, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:57,631 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085257631"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085257631"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085257631"}]},"ts":"1689085257631"} 2023-07-11 14:20:57,633 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=30, state=RUNNABLE; OpenRegionProcedure aa3553f00fe04c2619619d4fa27dd51d, server=jenkins-hbase9.apache.org,33099,1689085243944}] 2023-07-11 14:20:57,634 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=ce18e850fe81c0f0bf4b2c1436f84994, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:57,635 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085257634"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085257634"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085257634"}]},"ts":"1689085257634"} 2023-07-11 14:20:57,636 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=35, ppid=31, state=RUNNABLE; OpenRegionProcedure ce18e850fe81c0f0bf4b2c1436f84994, server=jenkins-hbase9.apache.org,33099,1689085243944}] 2023-07-11 14:20:57,777 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:57,777 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-11 14:20:57,779 INFO [RS-EventLoopGroup-14-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:48920, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-11 14:20:57,783 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. 2023-07-11 14:20:57,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ce18e850fe81c0f0bf4b2c1436f84994, NAME => 'testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994.', STARTKEY => 'c', ENDKEY => ''} 2023-07-11 14:20:57,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:20:57,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:57,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:20:57,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:20:57,785 INFO [StoreOpener-ce18e850fe81c0f0bf4b2c1436f84994-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:20:57,786 DEBUG [StoreOpener-ce18e850fe81c0f0bf4b2c1436f84994-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994/f1 2023-07-11 14:20:57,786 DEBUG [StoreOpener-ce18e850fe81c0f0bf4b2c1436f84994-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994/f1 2023-07-11 14:20:57,787 INFO [StoreOpener-ce18e850fe81c0f0bf4b2c1436f84994-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ce18e850fe81c0f0bf4b2c1436f84994 columnFamilyName f1 2023-07-11 14:20:57,787 INFO [StoreOpener-ce18e850fe81c0f0bf4b2c1436f84994-1] regionserver.HStore(310): Store=ce18e850fe81c0f0bf4b2c1436f84994/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:57,788 INFO [StoreOpener-ce18e850fe81c0f0bf4b2c1436f84994-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:20:57,789 DEBUG [StoreOpener-ce18e850fe81c0f0bf4b2c1436f84994-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994/f2 2023-07-11 14:20:57,789 DEBUG [StoreOpener-ce18e850fe81c0f0bf4b2c1436f84994-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994/f2 2023-07-11 14:20:57,790 INFO [StoreOpener-ce18e850fe81c0f0bf4b2c1436f84994-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ce18e850fe81c0f0bf4b2c1436f84994 columnFamilyName f2 2023-07-11 14:20:57,790 INFO [StoreOpener-ce18e850fe81c0f0bf4b2c1436f84994-1] regionserver.HStore(310): Store=ce18e850fe81c0f0bf4b2c1436f84994/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:57,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:20:57,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:20:57,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:20:57,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:20:57,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:57,798 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened ce18e850fe81c0f0bf4b2c1436f84994; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69361313, jitterRate=0.03356410562992096}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:20:57,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for ce18e850fe81c0f0bf4b2c1436f84994: 2023-07-11 14:20:57,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994., pid=35, masterSystemTime=1689085257777 2023-07-11 14:20:57,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. 2023-07-11 14:20:57,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. 2023-07-11 14:20:57,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. 2023-07-11 14:20:57,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => aa3553f00fe04c2619619d4fa27dd51d, NAME => 'testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d.', STARTKEY => 'b', ENDKEY => 'c'} 2023-07-11 14:20:57,802 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=ce18e850fe81c0f0bf4b2c1436f84994, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:57,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:20:57,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:57,802 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085257802"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085257802"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085257802"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085257802"}]},"ts":"1689085257802"} 2023-07-11 14:20:57,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:20:57,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:20:57,804 INFO [StoreOpener-aa3553f00fe04c2619619d4fa27dd51d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:20:57,805 DEBUG [StoreOpener-aa3553f00fe04c2619619d4fa27dd51d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d/f1 2023-07-11 14:20:57,806 DEBUG [StoreOpener-aa3553f00fe04c2619619d4fa27dd51d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d/f1 2023-07-11 14:20:57,806 INFO [StoreOpener-aa3553f00fe04c2619619d4fa27dd51d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aa3553f00fe04c2619619d4fa27dd51d columnFamilyName f1 2023-07-11 14:20:57,806 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=35, resume processing ppid=31 2023-07-11 14:20:57,806 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, ppid=31, state=SUCCESS; OpenRegionProcedure ce18e850fe81c0f0bf4b2c1436f84994, server=jenkins-hbase9.apache.org,33099,1689085243944 in 168 msec 2023-07-11 14:20:57,807 INFO [StoreOpener-aa3553f00fe04c2619619d4fa27dd51d-1] regionserver.HStore(310): Store=aa3553f00fe04c2619619d4fa27dd51d/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:57,807 INFO [StoreOpener-aa3553f00fe04c2619619d4fa27dd51d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:20:57,808 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=ce18e850fe81c0f0bf4b2c1436f84994, ASSIGN in 338 msec 2023-07-11 14:20:57,808 DEBUG [StoreOpener-aa3553f00fe04c2619619d4fa27dd51d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d/f2 2023-07-11 14:20:57,808 DEBUG [StoreOpener-aa3553f00fe04c2619619d4fa27dd51d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d/f2 2023-07-11 14:20:57,808 INFO [StoreOpener-aa3553f00fe04c2619619d4fa27dd51d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aa3553f00fe04c2619619d4fa27dd51d columnFamilyName f2 2023-07-11 14:20:57,809 INFO [StoreOpener-aa3553f00fe04c2619619d4fa27dd51d-1] regionserver.HStore(310): Store=aa3553f00fe04c2619619d4fa27dd51d/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:57,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:20:57,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:20:57,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:20:57,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:20:57,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:57,816 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened aa3553f00fe04c2619619d4fa27dd51d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67618086, jitterRate=0.007587999105453491}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:20:57,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for aa3553f00fe04c2619619d4fa27dd51d: 2023-07-11 14:20:57,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d., pid=34, masterSystemTime=1689085257777 2023-07-11 14:20:57,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. 2023-07-11 14:20:57,818 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. 2023-07-11 14:20:57,818 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. 2023-07-11 14:20:57,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a160933024ed654b6b55eac434a2d2d1, NAME => 'testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1.', STARTKEY => 'a', ENDKEY => 'b'} 2023-07-11 14:20:57,819 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=aa3553f00fe04c2619619d4fa27dd51d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:57,819 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085257819"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085257819"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085257819"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085257819"}]},"ts":"1689085257819"} 2023-07-11 14:20:57,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 a160933024ed654b6b55eac434a2d2d1 2023-07-11 14:20:57,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:57,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for a160933024ed654b6b55eac434a2d2d1 2023-07-11 14:20:57,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for a160933024ed654b6b55eac434a2d2d1 2023-07-11 14:20:57,820 INFO [StoreOpener-a160933024ed654b6b55eac434a2d2d1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region a160933024ed654b6b55eac434a2d2d1 2023-07-11 14:20:57,822 DEBUG [StoreOpener-a160933024ed654b6b55eac434a2d2d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1/f1 2023-07-11 14:20:57,822 DEBUG [StoreOpener-a160933024ed654b6b55eac434a2d2d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1/f1 2023-07-11 14:20:57,822 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=30 2023-07-11 14:20:57,822 INFO [StoreOpener-a160933024ed654b6b55eac434a2d2d1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a160933024ed654b6b55eac434a2d2d1 columnFamilyName f1 2023-07-11 14:20:57,822 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=30, state=SUCCESS; OpenRegionProcedure aa3553f00fe04c2619619d4fa27dd51d, server=jenkins-hbase9.apache.org,33099,1689085243944 in 187 msec 2023-07-11 14:20:57,823 INFO [StoreOpener-a160933024ed654b6b55eac434a2d2d1-1] regionserver.HStore(310): Store=a160933024ed654b6b55eac434a2d2d1/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:57,823 INFO [StoreOpener-a160933024ed654b6b55eac434a2d2d1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region a160933024ed654b6b55eac434a2d2d1 2023-07-11 14:20:57,823 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=aa3553f00fe04c2619619d4fa27dd51d, ASSIGN in 354 msec 2023-07-11 14:20:57,824 DEBUG [StoreOpener-a160933024ed654b6b55eac434a2d2d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1/f2 2023-07-11 14:20:57,824 DEBUG [StoreOpener-a160933024ed654b6b55eac434a2d2d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1/f2 2023-07-11 14:20:57,824 INFO [StoreOpener-a160933024ed654b6b55eac434a2d2d1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a160933024ed654b6b55eac434a2d2d1 columnFamilyName f2 2023-07-11 14:20:57,825 INFO [StoreOpener-a160933024ed654b6b55eac434a2d2d1-1] regionserver.HStore(310): Store=a160933024ed654b6b55eac434a2d2d1/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:57,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1 2023-07-11 14:20:57,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1 2023-07-11 14:20:57,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:20:57,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for a160933024ed654b6b55eac434a2d2d1 2023-07-11 14:20:57,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:57,832 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened a160933024ed654b6b55eac434a2d2d1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62960672, jitterRate=-0.0618128776550293}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:20:57,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for a160933024ed654b6b55eac434a2d2d1: 2023-07-11 14:20:57,832 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1., pid=33, masterSystemTime=1689085257777 2023-07-11 14:20:57,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. 2023-07-11 14:20:57,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. 2023-07-11 14:20:57,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. 2023-07-11 14:20:57,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9273ec20102dfbe67a6417ad2c513b65, NAME => 'testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65.', STARTKEY => '', ENDKEY => 'a'} 2023-07-11 14:20:57,834 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=a160933024ed654b6b55eac434a2d2d1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:57,834 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689085257834"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085257834"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085257834"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085257834"}]},"ts":"1689085257834"} 2023-07-11 14:20:57,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 9273ec20102dfbe67a6417ad2c513b65 2023-07-11 14:20:57,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:20:57,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 9273ec20102dfbe67a6417ad2c513b65 2023-07-11 14:20:57,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 9273ec20102dfbe67a6417ad2c513b65 2023-07-11 14:20:57,835 INFO [StoreOpener-9273ec20102dfbe67a6417ad2c513b65-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 9273ec20102dfbe67a6417ad2c513b65 2023-07-11 14:20:57,837 DEBUG [StoreOpener-9273ec20102dfbe67a6417ad2c513b65-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65/f1 2023-07-11 14:20:57,837 DEBUG [StoreOpener-9273ec20102dfbe67a6417ad2c513b65-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65/f1 2023-07-11 14:20:57,837 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=29 2023-07-11 14:20:57,837 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=29, state=SUCCESS; OpenRegionProcedure a160933024ed654b6b55eac434a2d2d1, server=jenkins-hbase9.apache.org,33099,1689085243944 in 208 msec 2023-07-11 14:20:57,837 INFO [StoreOpener-9273ec20102dfbe67a6417ad2c513b65-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9273ec20102dfbe67a6417ad2c513b65 columnFamilyName f1 2023-07-11 14:20:57,838 INFO [StoreOpener-9273ec20102dfbe67a6417ad2c513b65-1] regionserver.HStore(310): Store=9273ec20102dfbe67a6417ad2c513b65/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:57,838 INFO [StoreOpener-9273ec20102dfbe67a6417ad2c513b65-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 9273ec20102dfbe67a6417ad2c513b65 2023-07-11 14:20:57,838 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=a160933024ed654b6b55eac434a2d2d1, ASSIGN in 369 msec 2023-07-11 14:20:57,839 DEBUG [StoreOpener-9273ec20102dfbe67a6417ad2c513b65-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65/f2 2023-07-11 14:20:57,839 DEBUG [StoreOpener-9273ec20102dfbe67a6417ad2c513b65-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65/f2 2023-07-11 14:20:57,840 INFO [StoreOpener-9273ec20102dfbe67a6417ad2c513b65-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9273ec20102dfbe67a6417ad2c513b65 columnFamilyName f2 2023-07-11 14:20:57,840 INFO [StoreOpener-9273ec20102dfbe67a6417ad2c513b65-1] regionserver.HStore(310): Store=9273ec20102dfbe67a6417ad2c513b65/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:20:57,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65 2023-07-11 14:20:57,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65 2023-07-11 14:20:57,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:20:57,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 9273ec20102dfbe67a6417ad2c513b65 2023-07-11 14:20:57,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:20:57,846 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 9273ec20102dfbe67a6417ad2c513b65; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70273026, jitterRate=0.04714968800544739}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:20:57,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 9273ec20102dfbe67a6417ad2c513b65: 2023-07-11 14:20:57,846 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65., pid=32, masterSystemTime=1689085257777 2023-07-11 14:20:57,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. 2023-07-11 14:20:57,847 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. 2023-07-11 14:20:57,848 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=9273ec20102dfbe67a6417ad2c513b65, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:57,848 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689085257848"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085257848"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085257848"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085257848"}]},"ts":"1689085257848"} 2023-07-11 14:20:57,850 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=28 2023-07-11 14:20:57,851 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=28, state=SUCCESS; OpenRegionProcedure 9273ec20102dfbe67a6417ad2c513b65, server=jenkins-hbase9.apache.org,33099,1689085243944 in 225 msec 2023-07-11 14:20:57,852 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=27 2023-07-11 14:20:57,852 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=9273ec20102dfbe67a6417ad2c513b65, ASSIGN in 383 msec 2023-07-11 14:20:57,852 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085257852"}]},"ts":"1689085257852"} 2023-07-11 14:20:57,853 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=ENABLED in hbase:meta 2023-07-11 14:20:57,855 DEBUG [PEWorker-1] procedure.TruncateTableProcedure(145): truncate 'testTruncateWithFailoverAtStep4' completed 2023-07-11 14:20:57,856 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) in 4.9220 sec 2023-07-11 14:20:57,989 WARN [Listener at localhost/45445] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-11 14:20:57,989 DEBUG [Listener at localhost/45445] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testTruncateWithFailoverAtStep4 get assigned. Timeout = 60000ms 2023-07-11 14:20:57,991 INFO [Listener at localhost/45445] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-11 14:20:57,994 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(3484): All regions for table testTruncateWithFailoverAtStep4 assigned to meta. Checking AM states. 2023-07-11 14:20:57,995 INFO [Listener at localhost/45445] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-11 14:20:57,995 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(3504): All regions for table testTruncateWithFailoverAtStep4 assigned. 2023-07-11 14:20:57,999 DEBUG [Listener at localhost/45445] util.CommonFSUtils(682): File system contents for path hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4 2023-07-11 14:20:57,999 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |-.tabledesc/ 2023-07-11 14:20:58,000 DEBUG [Listener at localhost/45445] util.CommonFSUtils(702): |----.tableinfo.0000000001 2023-07-11 14:20:58,000 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |-.tmp/ 2023-07-11 14:20:58,000 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |-9273ec20102dfbe67a6417ad2c513b65/ 2023-07-11 14:20:58,000 DEBUG [Listener at localhost/45445] util.CommonFSUtils(702): |----.regioninfo 2023-07-11 14:20:58,001 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |----f1/ 2023-07-11 14:20:58,001 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |----f2/ 2023-07-11 14:20:58,001 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-11 14:20:58,002 DEBUG [Listener at localhost/45445] util.CommonFSUtils(702): |-------1.seqid 2023-07-11 14:20:58,002 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |-a160933024ed654b6b55eac434a2d2d1/ 2023-07-11 14:20:58,002 DEBUG [Listener at localhost/45445] util.CommonFSUtils(702): |----.regioninfo 2023-07-11 14:20:58,002 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |----f1/ 2023-07-11 14:20:58,002 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |----f2/ 2023-07-11 14:20:58,003 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-11 14:20:58,003 DEBUG [Listener at localhost/45445] util.CommonFSUtils(702): |-------1.seqid 2023-07-11 14:20:58,003 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |-aa3553f00fe04c2619619d4fa27dd51d/ 2023-07-11 14:20:58,004 DEBUG [Listener at localhost/45445] util.CommonFSUtils(702): |----.regioninfo 2023-07-11 14:20:58,004 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |----f1/ 2023-07-11 14:20:58,004 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |----f2/ 2023-07-11 14:20:58,004 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-11 14:20:58,005 DEBUG [Listener at localhost/45445] util.CommonFSUtils(702): |-------1.seqid 2023-07-11 14:20:58,005 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |-ce18e850fe81c0f0bf4b2c1436f84994/ 2023-07-11 14:20:58,005 DEBUG [Listener at localhost/45445] util.CommonFSUtils(702): |----.regioninfo 2023-07-11 14:20:58,005 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |----f1/ 2023-07-11 14:20:58,006 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |----f2/ 2023-07-11 14:20:58,006 DEBUG [Listener at localhost/45445] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-11 14:20:58,006 DEBUG [Listener at localhost/45445] util.CommonFSUtils(702): |-------1.seqid 2023-07-11 14:20:58,013 DEBUG [Listener at localhost/45445] procedure.MasterProcedureTestingUtility(212): Table directory layout is as expected. 2023-07-11 14:20:58,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33099] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Scan size: 154 connection: 172.31.2.10:40718 deadline: 1689085318018, exception=org.apache.hadoop.hbase.NotServingRegionException: testTruncateWithFailoverAtStep4,,1689085247772.15ef30302259990af4c93ea5956e8be7. is not online on jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:58,126 DEBUG [Listener at localhost/45445] client.ClientScanner(240): Advancing internal scanner to startKey at 'a', inclusive 2023-07-11 14:20:58,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33099] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Scan size: 158 connection: 172.31.2.10:40718 deadline: 1689085318127, exception=org.apache.hadoop.hbase.NotServingRegionException: testTruncateWithFailoverAtStep4,a,1689085247772.deb98f3ca0218078c17361c806c6f249. is not online on jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:58,233 DEBUG [Listener at localhost/45445] client.ClientScanner(240): Advancing internal scanner to startKey at 'b', inclusive 2023-07-11 14:20:58,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33099] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Scan size: 158 connection: 172.31.2.10:40718 deadline: 1689085318234, exception=org.apache.hadoop.hbase.NotServingRegionException: testTruncateWithFailoverAtStep4,b,1689085247772.c887ddcecdd2e29691c3631bd0b10745. is not online on jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:58,340 DEBUG [Listener at localhost/45445] client.ClientScanner(240): Advancing internal scanner to startKey at 'c', inclusive 2023-07-11 14:20:58,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33099] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Scan size: 158 connection: 172.31.2.10:40718 deadline: 1689085318341, exception=org.apache.hadoop.hbase.NotServingRegionException: testTruncateWithFailoverAtStep4,c,1689085247772.6c361c95cb323d45992c0bfbbe2c6f34. is not online on jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:58,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33099] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. with WAL disabled. Data may be lost in the event of a crash. 2023-07-11 14:20:58,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33099] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. with WAL disabled. Data may be lost in the event of a crash. 2023-07-11 14:20:58,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33099] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. with WAL disabled. Data may be lost in the event of a crash. 2023-07-11 14:20:58,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33099] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. with WAL disabled. Data may be lost in the event of a crash. 2023-07-11 14:20:58,456 DEBUG [Listener at localhost/45445] client.ClientScanner(240): Advancing internal scanner to startKey at 'a', inclusive 2023-07-11 14:20:58,458 DEBUG [Listener at localhost/45445] client.ClientScanner(240): Advancing internal scanner to startKey at 'b', inclusive 2023-07-11 14:20:58,459 DEBUG [Listener at localhost/45445] client.ClientScanner(240): Advancing internal scanner to startKey at 'c', inclusive 2023-07-11 14:20:58,460 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-11 14:20:58,460 INFO [Listener at localhost/45445] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-11 14:20:58,460 DEBUG [Listener at localhost/45445] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a665a39 to 127.0.0.1:50076 2023-07-11 14:20:58,460 DEBUG [Listener at localhost/45445] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:58,461 DEBUG [Listener at localhost/45445] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-11 14:20:58,461 DEBUG [Listener at localhost/45445] util.JVMClusterUtil(257): Found active master hash=689287654, stopped=false 2023-07-11 14:20:58,461 INFO [Listener at localhost/45445] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase9.apache.org,38655,1689085243913 2023-07-11 14:20:58,463 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): regionserver:33099-0x10154b433ed0002, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:58,463 INFO [Listener at localhost/45445] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:20:58,463 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-11 14:20:58,465 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:20:58,465 DEBUG [Listener at localhost/45445] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43560ec4 to 127.0.0.1:50076 2023-07-11 14:20:58,466 DEBUG [Listener at localhost/45445] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:58,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33099-0x10154b433ed0002, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:58,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:20:58,466 INFO [Listener at localhost/45445] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,33099,1689085243944' ***** 2023-07-11 14:20:58,466 INFO [Listener at localhost/45445] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-11 14:20:58,466 INFO [RS:0;jenkins-hbase9:33099] regionserver.HeapMemoryManager(220): Stopping 2023-07-11 14:20:58,466 INFO [RS:0;jenkins-hbase9:33099] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-11 14:20:58,466 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-11 14:20:58,466 INFO [RS:0;jenkins-hbase9:33099] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-11 14:20:58,467 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(3305): Received CLOSE for 9273ec20102dfbe67a6417ad2c513b65 2023-07-11 14:20:58,467 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(3305): Received CLOSE for d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:58,467 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(3305): Received CLOSE for a160933024ed654b6b55eac434a2d2d1 2023-07-11 14:20:58,468 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 9273ec20102dfbe67a6417ad2c513b65, disabling compactions & flushes 2023-07-11 14:20:58,468 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(3305): Received CLOSE for ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:20:58,468 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. 2023-07-11 14:20:58,468 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(3305): Received CLOSE for aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:20:58,468 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. 2023-07-11 14:20:58,468 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1144): stopping server jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:20:58,468 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. after waiting 0 ms 2023-07-11 14:20:58,468 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. 2023-07-11 14:20:58,468 DEBUG [RS:0;jenkins-hbase9:33099] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31cc45fc to 127.0.0.1:50076 2023-07-11 14:20:58,468 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 9273ec20102dfbe67a6417ad2c513b65 2/2 column families, dataSize=4.32 KB heapSize=9.78 KB 2023-07-11 14:20:58,468 DEBUG [RS:0;jenkins-hbase9:33099] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:20:58,468 INFO [RS:0;jenkins-hbase9:33099] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-11 14:20:58,468 INFO [RS:0;jenkins-hbase9:33099] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-11 14:20:58,468 INFO [RS:0;jenkins-hbase9:33099] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-11 14:20:58,468 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-11 14:20:58,469 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1474): Waiting on 6 regions to close 2023-07-11 14:20:58,469 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1478): Online Regions={9273ec20102dfbe67a6417ad2c513b65=testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65., d0027ffc5e8dfd7f30f021f0fdb84921=hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921., 1588230740=hbase:meta,,1.1588230740, a160933024ed654b6b55eac434a2d2d1=testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1., ce18e850fe81c0f0bf4b2c1436f84994=testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994., aa3553f00fe04c2619619d4fa27dd51d=testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d.} 2023-07-11 14:20:58,469 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-11 14:20:58,469 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on 1588230740, 9273ec20102dfbe67a6417ad2c513b65, a160933024ed654b6b55eac434a2d2d1, aa3553f00fe04c2619619d4fa27dd51d, ce18e850fe81c0f0bf4b2c1436f84994, d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:58,469 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-11 14:20:58,469 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-11 14:20:58,469 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-11 14:20:58,469 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-11 14:20:58,469 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=18.80 KB heapSize=30.18 KB 2023-07-11 14:20:58,478 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.31 KB at sequenceid=47 (bloomFilter=false), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/.tmp/info/6456591af24e42b782cbf602d34801a3 2023-07-11 14:20:58,484 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 6456591af24e42b782cbf602d34801a3 2023-07-11 14:20:58,502 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-11 14:20:58,669 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on 1588230740, 9273ec20102dfbe67a6417ad2c513b65, a160933024ed654b6b55eac434a2d2d1, aa3553f00fe04c2619619d4fa27dd51d, ce18e850fe81c0f0bf4b2c1436f84994, d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:58,869 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on 1588230740, 9273ec20102dfbe67a6417ad2c513b65, a160933024ed654b6b55eac434a2d2d1, aa3553f00fe04c2619619d4fa27dd51d, ce18e850fe81c0f0bf4b2c1436f84994, d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:58,877 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.16 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65/.tmp/f1/467881e12cba4d88855b70d63c685475 2023-07-11 14:20:58,891 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=447 B at sequenceid=47 (bloomFilter=false), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/.tmp/rep_barrier/d21ec5a4a56345b3a4e4bd7e141326a6 2023-07-11 14:20:58,895 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d21ec5a4a56345b3a4e4bd7e141326a6 2023-07-11 14:20:59,070 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on 1588230740, 9273ec20102dfbe67a6417ad2c513b65, a160933024ed654b6b55eac434a2d2d1, aa3553f00fe04c2619619d4fa27dd51d, ce18e850fe81c0f0bf4b2c1436f84994, d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:59,270 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on 1588230740, 9273ec20102dfbe67a6417ad2c513b65, a160933024ed654b6b55eac434a2d2d1, aa3553f00fe04c2619619d4fa27dd51d, ce18e850fe81c0f0bf4b2c1436f84994, d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:59,289 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.16 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65/.tmp/f2/26d25b3bc71e47e9869a143a255460cc 2023-07-11 14:20:59,295 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65/.tmp/f1/467881e12cba4d88855b70d63c685475 as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65/f1/467881e12cba4d88855b70d63c685475 2023-07-11 14:20:59,300 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65/f1/467881e12cba4d88855b70d63c685475, entries=33, sequenceid=5, filesize=7.1 K 2023-07-11 14:20:59,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65/.tmp/f2/26d25b3bc71e47e9869a143a255460cc as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65/f2/26d25b3bc71e47e9869a143a255460cc 2023-07-11 14:20:59,302 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.05 KB at sequenceid=47 (bloomFilter=false), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/.tmp/table/8d4775f456b64c31acb57590888f0316 2023-07-11 14:20:59,306 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65/f2/26d25b3bc71e47e9869a143a255460cc, entries=33, sequenceid=5, filesize=7.1 K 2023-07-11 14:20:59,307 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8d4775f456b64c31acb57590888f0316 2023-07-11 14:20:59,308 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/.tmp/info/6456591af24e42b782cbf602d34801a3 as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/info/6456591af24e42b782cbf602d34801a3 2023-07-11 14:20:59,309 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~4.32 KB/4422, heapSize ~9.75 KB/9984, currentSize=0 B/0 for 9273ec20102dfbe67a6417ad2c513b65 in 841ms, sequenceid=5, compaction requested=false 2023-07-11 14:20:59,309 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testTruncateWithFailoverAtStep4' 2023-07-11 14:20:59,316 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 6456591af24e42b782cbf602d34801a3 2023-07-11 14:20:59,316 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/9273ec20102dfbe67a6417ad2c513b65/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-11 14:20:59,316 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/info/6456591af24e42b782cbf602d34801a3, entries=58, sequenceid=47, filesize=12.6 K 2023-07-11 14:20:59,317 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. 2023-07-11 14:20:59,317 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 9273ec20102dfbe67a6417ad2c513b65: 2023-07-11 14:20:59,317 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed testTruncateWithFailoverAtStep4,,1689085253473.9273ec20102dfbe67a6417ad2c513b65. 2023-07-11 14:20:59,317 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing d0027ffc5e8dfd7f30f021f0fdb84921, disabling compactions & flushes 2023-07-11 14:20:59,317 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921. 2023-07-11 14:20:59,317 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/.tmp/rep_barrier/d21ec5a4a56345b3a4e4bd7e141326a6 as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/rep_barrier/d21ec5a4a56345b3a4e4bd7e141326a6 2023-07-11 14:20:59,317 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921. 2023-07-11 14:20:59,317 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921. after waiting 0 ms 2023-07-11 14:20:59,318 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921. 2023-07-11 14:20:59,318 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing d0027ffc5e8dfd7f30f021f0fdb84921 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-11 14:20:59,323 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d21ec5a4a56345b3a4e4bd7e141326a6 2023-07-11 14:20:59,323 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/rep_barrier/d21ec5a4a56345b3a4e4bd7e141326a6, entries=4, sequenceid=47, filesize=5.4 K 2023-07-11 14:20:59,324 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/.tmp/table/8d4775f456b64c31acb57590888f0316 as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/table/8d4775f456b64c31acb57590888f0316 2023-07-11 14:20:59,329 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8d4775f456b64c31acb57590888f0316 2023-07-11 14:20:59,329 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/table/8d4775f456b64c31acb57590888f0316, entries=11, sequenceid=47, filesize=5.7 K 2023-07-11 14:20:59,330 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~18.80 KB/19256, heapSize ~30.13 KB/30856, currentSize=0 B/0 for 1588230740 in 861ms, sequenceid=47, compaction requested=false 2023-07-11 14:20:59,337 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/meta/1588230740/recovered.edits/50.seqid, newMaxSeqId=50, maxSeqId=1 2023-07-11 14:20:59,338 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-11 14:20:59,338 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-11 14:20:59,338 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-11 14:20:59,338 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-11 14:20:59,392 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-11 14:20:59,392 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-11 14:20:59,470 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1474): Waiting on 4 regions to close 2023-07-11 14:20:59,470 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1478): Online Regions={d0027ffc5e8dfd7f30f021f0fdb84921=hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921., a160933024ed654b6b55eac434a2d2d1=testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1., ce18e850fe81c0f0bf4b2c1436f84994=testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994., aa3553f00fe04c2619619d4fa27dd51d=testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d.} 2023-07-11 14:20:59,470 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on a160933024ed654b6b55eac434a2d2d1, aa3553f00fe04c2619619d4fa27dd51d, ce18e850fe81c0f0bf4b2c1436f84994, d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:59,671 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on a160933024ed654b6b55eac434a2d2d1, aa3553f00fe04c2619619d4fa27dd51d, ce18e850fe81c0f0bf4b2c1436f84994, d0027ffc5e8dfd7f30f021f0fdb84921 2023-07-11 14:20:59,733 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/namespace/d0027ffc5e8dfd7f30f021f0fdb84921/.tmp/info/b4232a7c394c4198bb23325162fa7248 2023-07-11 14:20:59,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/namespace/d0027ffc5e8dfd7f30f021f0fdb84921/.tmp/info/b4232a7c394c4198bb23325162fa7248 as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/namespace/d0027ffc5e8dfd7f30f021f0fdb84921/info/b4232a7c394c4198bb23325162fa7248 2023-07-11 14:20:59,743 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/namespace/d0027ffc5e8dfd7f30f021f0fdb84921/info/b4232a7c394c4198bb23325162fa7248, entries=2, sequenceid=6, filesize=4.8 K 2023-07-11 14:20:59,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for d0027ffc5e8dfd7f30f021f0fdb84921 in 426ms, sequenceid=6, compaction requested=false 2023-07-11 14:20:59,750 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/hbase/namespace/d0027ffc5e8dfd7f30f021f0fdb84921/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-11 14:20:59,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921. 2023-07-11 14:20:59,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for d0027ffc5e8dfd7f30f021f0fdb84921: 2023-07-11 14:20:59,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689085246497.d0027ffc5e8dfd7f30f021f0fdb84921. 2023-07-11 14:20:59,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing a160933024ed654b6b55eac434a2d2d1, disabling compactions & flushes 2023-07-11 14:20:59,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. 2023-07-11 14:20:59,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. 2023-07-11 14:20:59,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. after waiting 0 ms 2023-07-11 14:20:59,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. 2023-07-11 14:20:59,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing a160933024ed654b6b55eac434a2d2d1 2/2 column families, dataSize=532 B heapSize=1.63 KB 2023-07-11 14:20:59,871 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on a160933024ed654b6b55eac434a2d2d1, aa3553f00fe04c2619619d4fa27dd51d, ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:21:00,071 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on a160933024ed654b6b55eac434a2d2d1, aa3553f00fe04c2619619d4fa27dd51d, ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:21:00,164 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=266 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1/.tmp/f1/c4d8925127ca46dd8ac99340de147c3d 2023-07-11 14:21:00,271 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on a160933024ed654b6b55eac434a2d2d1, aa3553f00fe04c2619619d4fa27dd51d, ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:21:00,472 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-11 14:21:00,472 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1478): Online Regions={a160933024ed654b6b55eac434a2d2d1=testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1., ce18e850fe81c0f0bf4b2c1436f84994=testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994., aa3553f00fe04c2619619d4fa27dd51d=testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d.} 2023-07-11 14:21:00,472 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on a160933024ed654b6b55eac434a2d2d1, aa3553f00fe04c2619619d4fa27dd51d, ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:21:00,576 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=266 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1/.tmp/f2/e8eed38528f84f94bfee897cc57f5ae0 2023-07-11 14:21:00,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1/.tmp/f1/c4d8925127ca46dd8ac99340de147c3d as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1/f1/c4d8925127ca46dd8ac99340de147c3d 2023-07-11 14:21:00,587 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1/f1/c4d8925127ca46dd8ac99340de147c3d, entries=4, sequenceid=5, filesize=5.1 K 2023-07-11 14:21:00,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1/.tmp/f2/e8eed38528f84f94bfee897cc57f5ae0 as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1/f2/e8eed38528f84f94bfee897cc57f5ae0 2023-07-11 14:21:00,593 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1/f2/e8eed38528f84f94bfee897cc57f5ae0, entries=4, sequenceid=5, filesize=5.1 K 2023-07-11 14:21:00,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~532 B/532, heapSize ~1.59 KB/1632, currentSize=0 B/0 for a160933024ed654b6b55eac434a2d2d1 in 843ms, sequenceid=5, compaction requested=false 2023-07-11 14:21:00,600 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/a160933024ed654b6b55eac434a2d2d1/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-11 14:21:00,600 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. 2023-07-11 14:21:00,600 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for a160933024ed654b6b55eac434a2d2d1: 2023-07-11 14:21:00,600 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed testTruncateWithFailoverAtStep4,a,1689085253473.a160933024ed654b6b55eac434a2d2d1. 2023-07-11 14:21:00,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing ce18e850fe81c0f0bf4b2c1436f84994, disabling compactions & flushes 2023-07-11 14:21:00,601 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. 2023-07-11 14:21:00,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. 2023-07-11 14:21:00,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. after waiting 0 ms 2023-07-11 14:21:00,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. 2023-07-11 14:21:00,601 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing ce18e850fe81c0f0bf4b2c1436f84994 2/2 column families, dataSize=1.30 KB heapSize=3.31 KB 2023-07-11 14:21:00,672 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on aa3553f00fe04c2619619d4fa27dd51d, ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:21:00,872 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on aa3553f00fe04c2619619d4fa27dd51d, ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:21:01,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=668 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994/.tmp/f1/9a4547617dc646e2a413fcad1b14eddb 2023-07-11 14:21:01,072 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on aa3553f00fe04c2619619d4fa27dd51d, ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:21:01,273 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on aa3553f00fe04c2619619d4fa27dd51d, ce18e850fe81c0f0bf4b2c1436f84994 2023-07-11 14:21:01,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=668 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994/.tmp/f2/11d5430c34304054b85b5f34bdabd68b 2023-07-11 14:21:01,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994/.tmp/f1/9a4547617dc646e2a413fcad1b14eddb as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994/f1/9a4547617dc646e2a413fcad1b14eddb 2023-07-11 14:21:01,432 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994/f1/9a4547617dc646e2a413fcad1b14eddb, entries=10, sequenceid=5, filesize=5.5 K 2023-07-11 14:21:01,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994/.tmp/f2/11d5430c34304054b85b5f34bdabd68b as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994/f2/11d5430c34304054b85b5f34bdabd68b 2023-07-11 14:21:01,437 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994/f2/11d5430c34304054b85b5f34bdabd68b, entries=10, sequenceid=5, filesize=5.5 K 2023-07-11 14:21:01,438 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.30 KB/1336, heapSize ~3.28 KB/3360, currentSize=0 B/0 for ce18e850fe81c0f0bf4b2c1436f84994 in 837ms, sequenceid=5, compaction requested=false 2023-07-11 14:21:01,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/ce18e850fe81c0f0bf4b2c1436f84994/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-11 14:21:01,443 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. 2023-07-11 14:21:01,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for ce18e850fe81c0f0bf4b2c1436f84994: 2023-07-11 14:21:01,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed testTruncateWithFailoverAtStep4,c,1689085253473.ce18e850fe81c0f0bf4b2c1436f84994. 2023-07-11 14:21:01,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing aa3553f00fe04c2619619d4fa27dd51d, disabling compactions & flushes 2023-07-11 14:21:01,444 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. 2023-07-11 14:21:01,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. 2023-07-11 14:21:01,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. after waiting 0 ms 2023-07-11 14:21:01,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. 2023-07-11 14:21:01,444 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing aa3553f00fe04c2619619d4fa27dd51d 2/2 column families, dataSize=398 B heapSize=1.34 KB 2023-07-11 14:21:01,473 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-11 14:21:01,473 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1478): Online Regions={aa3553f00fe04c2619619d4fa27dd51d=testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d.} 2023-07-11 14:21:01,473 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:21:01,673 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:21:01,851 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=199 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d/.tmp/f1/7311b9147f2046549c1a932ab5212502 2023-07-11 14:21:01,873 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:21:02,074 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:21:02,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=199 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d/.tmp/f2/30a4bf7058574cc595274341dc75ccd5 2023-07-11 14:21:02,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d/.tmp/f1/7311b9147f2046549c1a932ab5212502 as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d/f1/7311b9147f2046549c1a932ab5212502 2023-07-11 14:21:02,274 DEBUG [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1504): Waiting on aa3553f00fe04c2619619d4fa27dd51d 2023-07-11 14:21:02,274 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d/f1/7311b9147f2046549c1a932ab5212502, entries=3, sequenceid=5, filesize=5.0 K 2023-07-11 14:21:02,275 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d/.tmp/f2/30a4bf7058574cc595274341dc75ccd5 as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d/f2/30a4bf7058574cc595274341dc75ccd5 2023-07-11 14:21:02,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d/f2/30a4bf7058574cc595274341dc75ccd5, entries=3, sequenceid=5, filesize=5.0 K 2023-07-11 14:21:02,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~398 B/398, heapSize ~1.31 KB/1344, currentSize=0 B/0 for aa3553f00fe04c2619619d4fa27dd51d in 838ms, sequenceid=5, compaction requested=false 2023-07-11 14:21:02,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/data/default/testTruncateWithFailoverAtStep4/aa3553f00fe04c2619619d4fa27dd51d/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-11 14:21:02,289 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. 2023-07-11 14:21:02,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for aa3553f00fe04c2619619d4fa27dd51d: 2023-07-11 14:21:02,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed testTruncateWithFailoverAtStep4,b,1689085253473.aa3553f00fe04c2619619d4fa27dd51d. 2023-07-11 14:21:02,474 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,33099,1689085243944; all regions closed. 2023-07-11 14:21:02,479 DEBUG [RS:0;jenkins-hbase9:33099] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/oldWALs 2023-07-11 14:21:02,479 INFO [RS:0;jenkins-hbase9:33099] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C33099%2C1689085243944.meta:.meta(num 1689085246439) 2023-07-11 14:21:02,483 DEBUG [RS:0;jenkins-hbase9:33099] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/oldWALs 2023-07-11 14:21:02,483 INFO [RS:0;jenkins-hbase9:33099] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C33099%2C1689085243944:(num 1689085245519) 2023-07-11 14:21:02,483 DEBUG [RS:0;jenkins-hbase9:33099] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:21:02,483 INFO [RS:0;jenkins-hbase9:33099] regionserver.LeaseManager(133): Closed leases 2023-07-11 14:21:02,483 INFO [RS:0;jenkins-hbase9:33099] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-11 14:21:02,484 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 14:21:02,484 INFO [RS:0;jenkins-hbase9:33099] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:33099 2023-07-11 14:21:02,486 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): regionserver:33099-0x10154b433ed0002, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,33099,1689085243944 2023-07-11 14:21:02,486 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 14:21:02,486 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): regionserver:33099-0x10154b433ed0002, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 14:21:02,488 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,33099,1689085243944] 2023-07-11 14:21:02,488 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,33099,1689085243944; numProcessing=1 2023-07-11 14:21:02,489 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,33099,1689085243944 already deleted, retry=false 2023-07-11 14:21:02,489 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,33099,1689085243944 expired; onlineServers=0 2023-07-11 14:21:02,489 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,38655,1689085243913' ***** 2023-07-11 14:21:02,489 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-11 14:21:02,490 DEBUG [M:1;jenkins-hbase9:38655] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d2a301c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-11 14:21:02,490 INFO [M:1;jenkins-hbase9:38655] regionserver.HRegionServer(1144): stopping server jenkins-hbase9.apache.org,38655,1689085243913 2023-07-11 14:21:02,490 INFO [M:1;jenkins-hbase9:38655] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,38655,1689085243913; all regions closed. 2023-07-11 14:21:02,490 DEBUG [M:1;jenkins-hbase9:38655] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:21:02,490 DEBUG [M:1;jenkins-hbase9:38655] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-11 14:21:02,490 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-11 14:21:02,490 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085257422] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085257422,5,FailOnTimeoutGroup] 2023-07-11 14:21:02,490 DEBUG [M:1;jenkins-hbase9:38655] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-11 14:21:02,490 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085257422] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085257422,5,FailOnTimeoutGroup] 2023-07-11 14:21:02,491 INFO [M:1;jenkins-hbase9:38655] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-11 14:21:02,491 INFO [M:1;jenkins-hbase9:38655] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-11 14:21:02,491 INFO [M:1;jenkins-hbase9:38655] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-07-11 14:21:02,491 DEBUG [M:1;jenkins-hbase9:38655] master.HMaster(1512): Stopping service threads 2023-07-11 14:21:02,491 INFO [M:1;jenkins-hbase9:38655] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-11 14:21:02,491 ERROR [M:1;jenkins-hbase9:38655] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:21:02,492 INFO [M:1;jenkins-hbase9:38655] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:21:02,492 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-11 14:21:02,492 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:21:02,492 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:02,492 DEBUG [M:1;jenkins-hbase9:38655] zookeeper.ZKUtil(398): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-11 14:21:02,492 WARN [M:1;jenkins-hbase9:38655] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-11 14:21:02,492 INFO [M:1;jenkins-hbase9:38655] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:21:02,493 INFO [M:1;jenkins-hbase9:38655] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-11 14:21:02,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:21:02,493 DEBUG [M:1;jenkins-hbase9:38655] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 14:21:02,493 INFO [M:1;jenkins-hbase9:38655] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:02,493 DEBUG [M:1;jenkins-hbase9:38655] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:02,493 DEBUG [M:1;jenkins-hbase9:38655] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 14:21:02,493 DEBUG [M:1;jenkins-hbase9:38655] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:02,493 INFO [M:1;jenkins-hbase9:38655] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=26.01 KB heapSize=31.70 KB 2023-07-11 14:21:02,588 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): regionserver:33099-0x10154b433ed0002, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:21:02,588 INFO [RS:0;jenkins-hbase9:33099] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,33099,1689085243944; zookeeper connection closed. 2023-07-11 14:21:02,588 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): regionserver:33099-0x10154b433ed0002, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:21:02,589 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2303f5f0] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2303f5f0 2023-07-11 14:21:02,589 INFO [Listener at localhost/45445] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 1 regionserver(s) complete 2023-07-11 14:21:02,901 INFO [M:1;jenkins-hbase9:38655] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.01 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b8c3ebc603744a8abc077911ec416fce 2023-07-11 14:21:02,907 DEBUG [M:1;jenkins-hbase9:38655] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b8c3ebc603744a8abc077911ec416fce as hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b8c3ebc603744a8abc077911ec416fce 2023-07-11 14:21:02,912 INFO [M:1;jenkins-hbase9:38655] regionserver.HStore(1080): Added hdfs://localhost:43147/user/jenkins/test-data/e07a3848-0b62-0152-a209-d5b50458586e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b8c3ebc603744a8abc077911ec416fce, entries=9, sequenceid=253, filesize=5.4 K 2023-07-11 14:21:02,913 INFO [M:1;jenkins-hbase9:38655] regionserver.HRegion(2948): Finished flush of dataSize ~26.01 KB/26634, heapSize ~31.68 KB/32440, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 420ms, sequenceid=253, compaction requested=false 2023-07-11 14:21:02,915 INFO [M:1;jenkins-hbase9:38655] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:02,915 DEBUG [M:1;jenkins-hbase9:38655] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:21:02,918 INFO [M:1;jenkins-hbase9:38655] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-11 14:21:02,918 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 14:21:02,919 INFO [M:1;jenkins-hbase9:38655] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:38655 2023-07-11 14:21:02,921 DEBUG [M:1;jenkins-hbase9:38655] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,38655,1689085243913 already deleted, retry=false 2023-07-11 14:21:03,023 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:21:03,023 INFO [M:1;jenkins-hbase9:38655] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,38655,1689085243913; zookeeper connection closed. 2023-07-11 14:21:03,023 DEBUG [Listener at localhost/45445-EventThread] zookeeper.ZKWatcher(600): master:38655-0x10154b433ed0001, quorum=127.0.0.1:50076, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:21:03,024 WARN [Listener at localhost/45445] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-11 14:21:03,028 INFO [Listener at localhost/45445] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-11 14:21:03,131 WARN [BP-787187594-172.31.2.10-1689085243061 heartbeating to localhost/127.0.0.1:43147] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-11 14:21:03,132 WARN [BP-787187594-172.31.2.10-1689085243061 heartbeating to localhost/127.0.0.1:43147] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-787187594-172.31.2.10-1689085243061 (Datanode Uuid 41c70276-9c23-4ed1-80d1-224e00e225f4) service to localhost/127.0.0.1:43147 2023-07-11 14:21:03,132 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/cluster_ef34d284-81d2-8651-5312-c5ca43158945/dfs/data/data1/current/BP-787187594-172.31.2.10-1689085243061] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-11 14:21:03,133 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/cluster_ef34d284-81d2-8651-5312-c5ca43158945/dfs/data/data2/current/BP-787187594-172.31.2.10-1689085243061] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-11 14:21:03,144 INFO [Listener at localhost/45445] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-11 14:21:03,255 INFO [Listener at localhost/45445] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-11 14:21:03,266 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-07-11 14:21:03,277 INFO [Listener at localhost/45445] hbase.ResourceChecker(175): after: master.procedure.TestMasterFailoverWithProcedures#testTruncateWithFailover Thread=123 (was 101) Potentially hanging thread: IPC Client (200715596) connection to localhost/127.0.0.1:43147 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:43147 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (200715596) connection to localhost/127.0.0.1:43147 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-12-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.3@localhost:43147 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/45445 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (200715596) connection to localhost/127.0.0.1:43147 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-12-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=482 (was 446) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=193 (was 260), ProcessCount=171 (was 171), AvailableMemoryMB=7294 (was 7472) 2023-07-11 14:21:03,285 INFO [Listener at localhost/45445] hbase.ResourceChecker(147): before: master.procedure.TestMasterFailoverWithProcedures#testEnableTableWithFailover Thread=123, OpenFileDescriptor=482, MaxFileDescriptor=60000, SystemLoadAverage=193, ProcessCount=171, AvailableMemoryMB=7294 2023-07-11 14:21:03,286 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=2, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-11 14:21:03,286 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-11 14:21:03,286 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/hadoop.log.dir so I do NOT create it in target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934 2023-07-11 14:21:03,286 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/db9a0792-2586-768e-6c32-903e6a5784aa/hadoop.tmp.dir so I do NOT create it in target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934 2023-07-11 14:21:03,286 INFO [Listener at localhost/45445] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/cluster_b3df6d01-63d8-53bf-20c3-0b45a541bf8e, deleteOnExit=true 2023-07-11 14:21:03,286 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-11 14:21:03,286 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/test.cache.data in system properties and HBase conf 2023-07-11 14:21:03,286 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/hadoop.tmp.dir in system properties and HBase conf 2023-07-11 14:21:03,286 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/hadoop.log.dir in system properties and HBase conf 2023-07-11 14:21:03,286 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-11 14:21:03,287 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-11 14:21:03,287 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-11 14:21:03,287 DEBUG [Listener at localhost/45445] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-11 14:21:03,287 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-11 14:21:03,287 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-11 14:21:03,287 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-11 14:21:03,287 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-11 14:21:03,287 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-11 14:21:03,287 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-11 14:21:03,287 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-11 14:21:03,288 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-11 14:21:03,288 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-11 14:21:03,288 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/nfs.dump.dir in system properties and HBase conf 2023-07-11 14:21:03,288 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/java.io.tmpdir in system properties and HBase conf 2023-07-11 14:21:03,288 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-11 14:21:03,288 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-11 14:21:03,288 INFO [Listener at localhost/45445] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-11 14:21:03,292 WARN [Listener at localhost/45445] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-11 14:21:03,292 WARN [Listener at localhost/45445] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-11 14:21:03,350 WARN [Listener at localhost/45445] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-11 14:21:03,352 INFO [Listener at localhost/45445] log.Slf4jLog(67): jetty-6.1.26 2023-07-11 14:21:03,356 INFO [Listener at localhost/45445] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/java.io.tmpdir/Jetty_localhost_33437_hdfs____22fpia/webapp 2023-07-11 14:21:03,449 INFO [Listener at localhost/45445] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33437 2023-07-11 14:21:03,453 WARN [Listener at localhost/45445] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-11 14:21:03,453 WARN [Listener at localhost/45445] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-11 14:21:03,493 WARN [Listener at localhost/43599] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-11 14:21:03,508 WARN [Listener at localhost/43599] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-11 14:21:03,510 WARN [Listener at localhost/43599] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-11 14:21:03,511 INFO [Listener at localhost/43599] log.Slf4jLog(67): jetty-6.1.26 2023-07-11 14:21:03,517 INFO [Listener at localhost/43599] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/java.io.tmpdir/Jetty_localhost_37801_datanode____.fsv0p0/webapp 2023-07-11 14:21:03,610 INFO [Listener at localhost/43599] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37801 2023-07-11 14:21:03,616 WARN [Listener at localhost/45629] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-11 14:21:03,713 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x302f1315aa53b287: Processing first storage report for DS-b87c366b-ca4b-427d-8f38-36079d5e4b95 from datanode 68c167b7-9d05-40f2-8dc8-414baa8e4181 2023-07-11 14:21:03,714 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x302f1315aa53b287: from storage DS-b87c366b-ca4b-427d-8f38-36079d5e4b95 node DatanodeRegistration(127.0.0.1:45335, datanodeUuid=68c167b7-9d05-40f2-8dc8-414baa8e4181, infoPort=39767, infoSecurePort=0, ipcPort=45629, storageInfo=lv=-57;cid=testClusterID;nsid=793450506;c=1689085263316), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-11 14:21:03,714 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x302f1315aa53b287: Processing first storage report for DS-1ee30a75-4372-46ba-b849-50c9ad3df97f from datanode 68c167b7-9d05-40f2-8dc8-414baa8e4181 2023-07-11 14:21:03,714 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x302f1315aa53b287: from storage DS-1ee30a75-4372-46ba-b849-50c9ad3df97f node DatanodeRegistration(127.0.0.1:45335, datanodeUuid=68c167b7-9d05-40f2-8dc8-414baa8e4181, infoPort=39767, infoSecurePort=0, ipcPort=45629, storageInfo=lv=-57;cid=testClusterID;nsid=793450506;c=1689085263316), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-11 14:21:03,722 DEBUG [Listener at localhost/45629] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934 2023-07-11 14:21:03,724 INFO [Listener at localhost/45629] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/cluster_b3df6d01-63d8-53bf-20c3-0b45a541bf8e/zookeeper_0, clientPort=55314, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/cluster_b3df6d01-63d8-53bf-20c3-0b45a541bf8e/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/cluster_b3df6d01-63d8-53bf-20c3-0b45a541bf8e/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-11 14:21:03,725 INFO [Listener at localhost/45629] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=55314 2023-07-11 14:21:03,725 INFO [Listener at localhost/45629] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:21:03,726 INFO [Listener at localhost/45629] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:21:04,135 INFO [Listener at localhost/45629] util.FSUtils(471): Created version file at hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5 with version=8 2023-07-11 14:21:04,135 INFO [Listener at localhost/45629] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost:46725/user/jenkins/test-data/60a07658-e13c-6a8b-eecb-945963b62975/hbase-staging 2023-07-11 14:21:04,137 INFO [Listener at localhost/45629] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-07-11 14:21:04,137 INFO [Listener at localhost/45629] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:21:04,138 INFO [Listener at localhost/45629] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 14:21:04,138 INFO [Listener at localhost/45629] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 14:21:04,138 INFO [Listener at localhost/45629] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:21:04,138 INFO [Listener at localhost/45629] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 14:21:04,138 INFO [Listener at localhost/45629] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 14:21:04,139 INFO [Listener at localhost/45629] ipc.NettyRpcServer(120): Bind to /172.31.2.10:40067 2023-07-11 14:21:04,139 INFO [Listener at localhost/45629] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:21:04,141 INFO [Listener at localhost/45629] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:21:04,142 INFO [Listener at localhost/45629] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40067 connecting to ZooKeeper ensemble=127.0.0.1:55314 2023-07-11 14:21:04,152 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:400670x0, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 14:21:04,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:40067-0x10154b4830b0000 connected 2023-07-11 14:21:04,166 DEBUG [Listener at localhost/45629] zookeeper.ZKUtil(164): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:21:04,167 DEBUG [Listener at localhost/45629] zookeeper.ZKUtil(164): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:21:04,167 DEBUG [Listener at localhost/45629] zookeeper.ZKUtil(164): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 14:21:04,167 DEBUG [Listener at localhost/45629] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40067 2023-07-11 14:21:04,168 DEBUG [Listener at localhost/45629] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40067 2023-07-11 14:21:04,168 DEBUG [Listener at localhost/45629] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40067 2023-07-11 14:21:04,168 DEBUG [Listener at localhost/45629] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40067 2023-07-11 14:21:04,168 DEBUG [Listener at localhost/45629] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40067 2023-07-11 14:21:04,169 INFO [Listener at localhost/45629] master.HMaster(444): hbase.rootdir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5, hbase.cluster.distributed=false 2023-07-11 14:21:04,170 INFO [Listener at localhost/45629] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-07-11 14:21:04,170 INFO [Listener at localhost/45629] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:21:04,170 INFO [Listener at localhost/45629] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 14:21:04,170 INFO [Listener at localhost/45629] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 14:21:04,170 INFO [Listener at localhost/45629] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:21:04,170 INFO [Listener at localhost/45629] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 14:21:04,170 INFO [Listener at localhost/45629] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 14:21:04,171 INFO [Listener at localhost/45629] ipc.NettyRpcServer(120): Bind to /172.31.2.10:40895 2023-07-11 14:21:04,171 INFO [Listener at localhost/45629] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:21:04,172 INFO [Listener at localhost/45629] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:21:04,173 INFO [Listener at localhost/45629] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40895 connecting to ZooKeeper ensemble=127.0.0.1:55314 2023-07-11 14:21:04,175 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:408950x0, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 14:21:04,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:40895-0x10154b4830b0001 connected 2023-07-11 14:21:04,178 DEBUG [Listener at localhost/45629] zookeeper.ZKUtil(164): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:21:04,179 DEBUG [Listener at localhost/45629] zookeeper.ZKUtil(164): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:21:04,179 DEBUG [Listener at localhost/45629] zookeeper.ZKUtil(164): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 14:21:04,179 DEBUG [Listener at localhost/45629] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40895 2023-07-11 14:21:04,179 DEBUG [Listener at localhost/45629] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40895 2023-07-11 14:21:04,180 DEBUG [Listener at localhost/45629] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40895 2023-07-11 14:21:04,180 DEBUG [Listener at localhost/45629] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40895 2023-07-11 14:21:04,180 DEBUG [Listener at localhost/45629] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40895 2023-07-11 14:21:04,180 INFO [Listener at localhost/45629] master.HMaster(444): hbase.rootdir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5, hbase.cluster.distributed=false 2023-07-11 14:21:04,192 INFO [Listener at localhost/45629] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-07-11 14:21:04,192 INFO [Listener at localhost/45629] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:21:04,192 INFO [Listener at localhost/45629] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-11 14:21:04,192 INFO [Listener at localhost/45629] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-11 14:21:04,193 INFO [Listener at localhost/45629] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-11 14:21:04,193 INFO [Listener at localhost/45629] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-11 14:21:04,193 INFO [Listener at localhost/45629] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-11 14:21:04,193 INFO [Listener at localhost/45629] ipc.NettyRpcServer(120): Bind to /172.31.2.10:46501 2023-07-11 14:21:04,193 INFO [Listener at localhost/45629] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-11 14:21:04,198 DEBUG [Listener at localhost/45629] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-11 14:21:04,198 INFO [Listener at localhost/45629] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:21:04,199 INFO [Listener at localhost/45629] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:21:04,200 INFO [Listener at localhost/45629] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46501 connecting to ZooKeeper ensemble=127.0.0.1:55314 2023-07-11 14:21:04,203 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): regionserver:465010x0, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-11 14:21:04,204 DEBUG [Listener at localhost/45629] zookeeper.ZKUtil(164): regionserver:465010x0, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:21:04,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:46501-0x10154b4830b0002 connected 2023-07-11 14:21:04,205 DEBUG [Listener at localhost/45629] zookeeper.ZKUtil(164): regionserver:46501-0x10154b4830b0002, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:21:04,205 DEBUG [Listener at localhost/45629] zookeeper.ZKUtil(164): regionserver:46501-0x10154b4830b0002, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-11 14:21:04,206 DEBUG [Listener at localhost/45629] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46501 2023-07-11 14:21:04,206 DEBUG [Listener at localhost/45629] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46501 2023-07-11 14:21:04,206 DEBUG [Listener at localhost/45629] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46501 2023-07-11 14:21:04,206 DEBUG [Listener at localhost/45629] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46501 2023-07-11 14:21:04,206 DEBUG [Listener at localhost/45629] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46501 2023-07-11 14:21:04,207 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,40067,1689085264137 2023-07-11 14:21:04,208 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,40895,1689085264170 2023-07-11 14:21:04,209 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:21:04,209 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:21:04,210 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,40067,1689085264137 2023-07-11 14:21:04,210 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,40895,1689085264170 2023-07-11 14:21:04,211 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:21:04,211 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): regionserver:46501-0x10154b4830b0002, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:21:04,211 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:21:04,211 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:04,211 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:04,212 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:21:04,212 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:21:04,214 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,40067,1689085264137 from backup master directory 2023-07-11 14:21:04,214 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase9.apache.org,40067,1689085264137; waiting to become the next active master 2023-07-11 14:21:04,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:21:04,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:21:04,215 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,40067,1689085264137 2023-07-11 14:21:04,215 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:21:04,215 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:21:04,215 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 14:21:04,216 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase9.apache.org,40067,1689085264137 2023-07-11 14:21:04,230 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/hbase.id with ID: 91e7143d-a1bb-4bef-9957-b7389d3f971f 2023-07-11 14:21:04,243 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:21:04,247 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:04,247 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:04,518 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(152): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-11 14:21:04,678 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x4aec6543 to 127.0.0.1:55314 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:21:04,682 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54af52ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:21:04,682 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-11 14:21:04,683 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-11 14:21:04,683 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:21:04,684 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store-tmp 2023-07-11 14:21:05,096 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:05,096 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 14:21:05,096 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:05,096 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:05,096 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 14:21:05,096 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:05,096 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:05,096 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:21:05,097 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/WALs/jenkins-hbase9.apache.org,40067,1689085264137 2023-07-11 14:21:05,099 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C40067%2C1689085264137, suffix=, logDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/WALs/jenkins-hbase9.apache.org,40067,1689085264137, archiveDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/oldWALs, maxLogs=10 2023-07-11 14:21:05,113 DEBUG [RS-EventLoopGroup-17-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45335,DS-b87c366b-ca4b-427d-8f38-36079d5e4b95,DISK] 2023-07-11 14:21:05,115 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/WALs/jenkins-hbase9.apache.org,40067,1689085264137/jenkins-hbase9.apache.org%2C40067%2C1689085264137.1689085265099 2023-07-11 14:21:05,115 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45335,DS-b87c366b-ca4b-427d-8f38-36079d5e4b95,DISK]] 2023-07-11 14:21:05,115 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:21:05,115 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:05,115 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:21:05,115 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:21:05,116 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:21:05,117 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-11 14:21:05,118 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-11 14:21:05,118 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:05,119 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:21:05,119 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:21:05,122 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:21:05,125 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:21:05,125 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71999009, jitterRate=0.07286883890628815}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 14:21:05,125 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:21:05,125 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-11 14:21:05,126 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:21:05,126 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:21:05,126 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:21:05,127 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:21:05,127 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-07-11 14:21:05,127 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-11 14:21:05,128 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-11 14:21:05,128 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-11 14:21:05,138 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-11 14:21:05,139 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-11 14:21:05,139 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-11 14:21:05,139 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-11 14:21:05,139 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-11 14:21:05,141 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:05,141 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:05,141 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-11 14:21:05,142 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-11 14:21:05,142 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-11 14:21:05,143 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): regionserver:46501-0x10154b4830b0002, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 14:21:05,143 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 14:21:05,143 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-11 14:21:05,144 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:05,144 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:05,144 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase9.apache.org,40067,1689085264137, sessionid=0x10154b4830b0000, setting cluster-up flag (Was=false) 2023-07-11 14:21:05,148 INFO [M:1;jenkins-hbase9:40895] regionserver.HRegionServer(951): ClusterId : 91e7143d-a1bb-4bef-9957-b7389d3f971f 2023-07-11 14:21:05,149 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:05,149 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:05,153 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-11 14:21:05,154 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,40067,1689085264137 2023-07-11 14:21:05,156 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:05,156 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:05,161 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-11 14:21:05,162 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,40067,1689085264137 2023-07-11 14:21:05,163 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.hbase-snapshot/.tmp 2023-07-11 14:21:05,164 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-11 14:21:05,164 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:21:05,164 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:21:05,164 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:21:05,164 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:21:05,164 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-07-11 14:21:05,165 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:21:05,165 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-11 14:21:05,165 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:21:05,165 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085295165 2023-07-11 14:21:05,165 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-11 14:21:05,166 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-11 14:21:05,166 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-11 14:21:05,166 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-11 14:21:05,166 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-11 14:21:05,166 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-11 14:21:05,167 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:05,167 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-11 14:21:05,167 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-11 14:21:05,167 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-11 14:21:05,168 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-11 14:21:05,168 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-11 14:21:05,168 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-11 14:21:05,168 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-11 14:21:05,168 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085265168,5,FailOnTimeoutGroup] 2023-07-11 14:21:05,168 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-11 14:21:05,168 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085265168,5,FailOnTimeoutGroup] 2023-07-11 14:21:05,168 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:05,168 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-11 14:21:05,168 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:05,169 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:05,209 INFO [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(951): ClusterId : 91e7143d-a1bb-4bef-9957-b7389d3f971f 2023-07-11 14:21:05,209 DEBUG [RS:0;jenkins-hbase9:46501] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-11 14:21:05,211 DEBUG [RS:0;jenkins-hbase9:46501] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-11 14:21:05,211 DEBUG [RS:0;jenkins-hbase9:46501] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-11 14:21:05,212 DEBUG [RS:0;jenkins-hbase9:46501] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-11 14:21:05,213 DEBUG [RS:0;jenkins-hbase9:46501] zookeeper.ReadOnlyZKClient(139): Connect 0x07f6df26 to 127.0.0.1:55314 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:21:05,217 DEBUG [RS:0;jenkins-hbase9:46501] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@93131a8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:21:05,217 DEBUG [RS:0;jenkins-hbase9:46501] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d0eb456, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-11 14:21:05,226 DEBUG [RS:0;jenkins-hbase9:46501] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase9:46501 2023-07-11 14:21:05,226 INFO [RS:0;jenkins-hbase9:46501] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-11 14:21:05,226 INFO [RS:0;jenkins-hbase9:46501] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-11 14:21:05,226 DEBUG [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(1022): About to register with Master. 2023-07-11 14:21:05,227 INFO [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase9.apache.org,40067,1689085264137 with isa=jenkins-hbase9.apache.org/172.31.2.10:46501, startcode=1689085264192 2023-07-11 14:21:05,227 DEBUG [RS:0;jenkins-hbase9:46501] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-11 14:21:05,229 INFO [RS-EventLoopGroup-15-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:53813, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-07-11 14:21:05,230 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40067] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:05,230 DEBUG [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5 2023-07-11 14:21:05,231 DEBUG [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:43599 2023-07-11 14:21:05,231 DEBUG [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-11 14:21:05,232 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 14:21:05,232 DEBUG [RS:0;jenkins-hbase9:46501] zookeeper.ZKUtil(162): regionserver:46501-0x10154b4830b0002, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:05,233 WARN [RS:0;jenkins-hbase9:46501] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 14:21:05,233 INFO [RS:0;jenkins-hbase9:46501] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:21:05,233 DEBUG [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(1948): logDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/WALs/jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:05,233 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,46501,1689085264192] 2023-07-11 14:21:05,236 DEBUG [RS:0;jenkins-hbase9:46501] zookeeper.ZKUtil(162): regionserver:46501-0x10154b4830b0002, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:05,237 DEBUG [RS:0;jenkins-hbase9:46501] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-11 14:21:05,237 INFO [RS:0;jenkins-hbase9:46501] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-11 14:21:05,238 INFO [RS:0;jenkins-hbase9:46501] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-11 14:21:05,239 INFO [RS:0;jenkins-hbase9:46501] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-11 14:21:05,239 INFO [RS:0;jenkins-hbase9:46501] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:05,239 INFO [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-11 14:21:05,240 INFO [RS:0;jenkins-hbase9:46501] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:05,240 DEBUG [RS:0;jenkins-hbase9:46501] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:21:05,240 DEBUG [RS:0;jenkins-hbase9:46501] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:21:05,240 DEBUG [RS:0;jenkins-hbase9:46501] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:21:05,241 DEBUG [RS:0;jenkins-hbase9:46501] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:21:05,241 DEBUG [RS:0;jenkins-hbase9:46501] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:21:05,241 DEBUG [RS:0;jenkins-hbase9:46501] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-11 14:21:05,241 DEBUG [RS:0;jenkins-hbase9:46501] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:21:05,241 DEBUG [RS:0;jenkins-hbase9:46501] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:21:05,241 DEBUG [RS:0;jenkins-hbase9:46501] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:21:05,241 DEBUG [RS:0;jenkins-hbase9:46501] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:21:05,244 INFO [RS:0;jenkins-hbase9:46501] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:05,244 INFO [RS:0;jenkins-hbase9:46501] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:05,244 INFO [RS:0;jenkins-hbase9:46501] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:05,256 INFO [RS:0;jenkins-hbase9:46501] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-11 14:21:05,256 INFO [RS:0;jenkins-hbase9:46501] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,46501,1689085264192-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:05,266 INFO [RS:0;jenkins-hbase9:46501] regionserver.Replication(203): jenkins-hbase9.apache.org,46501,1689085264192 started 2023-07-11 14:21:05,266 INFO [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(1637): Serving as jenkins-hbase9.apache.org,46501,1689085264192, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:46501, sessionid=0x10154b4830b0002 2023-07-11 14:21:05,267 DEBUG [RS:0;jenkins-hbase9:46501] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-11 14:21:05,267 DEBUG [RS:0;jenkins-hbase9:46501] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:05,267 DEBUG [RS:0;jenkins-hbase9:46501] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,46501,1689085264192' 2023-07-11 14:21:05,267 DEBUG [RS:0;jenkins-hbase9:46501] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-11 14:21:05,267 DEBUG [RS:0;jenkins-hbase9:46501] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-11 14:21:05,267 DEBUG [RS:0;jenkins-hbase9:46501] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-11 14:21:05,267 DEBUG [RS:0;jenkins-hbase9:46501] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-11 14:21:05,267 DEBUG [RS:0;jenkins-hbase9:46501] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:05,267 DEBUG [RS:0;jenkins-hbase9:46501] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,46501,1689085264192' 2023-07-11 14:21:05,267 DEBUG [RS:0;jenkins-hbase9:46501] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-11 14:21:05,268 DEBUG [RS:0;jenkins-hbase9:46501] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-11 14:21:05,268 DEBUG [RS:0;jenkins-hbase9:46501] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-11 14:21:05,268 INFO [RS:0;jenkins-hbase9:46501] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-11 14:21:05,268 INFO [RS:0;jenkins-hbase9:46501] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-11 14:21:05,370 INFO [RS:0;jenkins-hbase9:46501] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C46501%2C1689085264192, suffix=, logDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/WALs/jenkins-hbase9.apache.org,46501,1689085264192, archiveDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/oldWALs, maxLogs=32 2023-07-11 14:21:05,384 DEBUG [RS-EventLoopGroup-17-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45335,DS-b87c366b-ca4b-427d-8f38-36079d5e4b95,DISK] 2023-07-11 14:21:05,385 INFO [RS:0;jenkins-hbase9:46501] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/WALs/jenkins-hbase9.apache.org,46501,1689085264192/jenkins-hbase9.apache.org%2C46501%2C1689085264192.1689085265370 2023-07-11 14:21:05,386 DEBUG [RS:0;jenkins-hbase9:46501] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45335,DS-b87c366b-ca4b-427d-8f38-36079d5e4b95,DISK]] 2023-07-11 14:21:05,577 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-11 14:21:05,578 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-11 14:21:05,578 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5 2023-07-11 14:21:05,984 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:05,985 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-11 14:21:05,986 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/info 2023-07-11 14:21:05,987 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-11 14:21:05,987 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:05,987 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-11 14:21:05,989 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/rep_barrier 2023-07-11 14:21:05,989 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-11 14:21:05,989 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:05,990 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-11 14:21:05,991 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/table 2023-07-11 14:21:05,991 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-11 14:21:05,991 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:05,992 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740 2023-07-11 14:21:05,992 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740 2023-07-11 14:21:05,994 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-11 14:21:05,995 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-11 14:21:05,997 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:21:05,997 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67818276, jitterRate=0.010571062564849854}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-11 14:21:05,997 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-11 14:21:05,997 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-11 14:21:05,997 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-11 14:21:05,997 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-11 14:21:05,998 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-11 14:21:05,998 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-11 14:21:05,998 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-11 14:21:05,998 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-11 14:21:05,998 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-11 14:21:05,999 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-11 14:21:05,999 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-11 14:21:06,000 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-11 14:21:06,000 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-11 14:21:06,150 DEBUG [jenkins-hbase9:40067] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-11 14:21:06,151 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,46501,1689085264192, state=OPENING 2023-07-11 14:21:06,152 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-11 14:21:06,155 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:06,155 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:06,155 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:21:06,155 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:21:06,155 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,46501,1689085264192}] 2023-07-11 14:21:06,308 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:06,308 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-11 14:21:06,310 INFO [RS-EventLoopGroup-17-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:33122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-11 14:21:06,313 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-11 14:21:06,313 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:21:06,315 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C46501%2C1689085264192.meta, suffix=.meta, logDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/WALs/jenkins-hbase9.apache.org,46501,1689085264192, archiveDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/oldWALs, maxLogs=32 2023-07-11 14:21:06,328 DEBUG [RS-EventLoopGroup-17-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45335,DS-b87c366b-ca4b-427d-8f38-36079d5e4b95,DISK] 2023-07-11 14:21:06,330 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/WALs/jenkins-hbase9.apache.org,46501,1689085264192/jenkins-hbase9.apache.org%2C46501%2C1689085264192.meta.1689085266315.meta 2023-07-11 14:21:06,330 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45335,DS-b87c366b-ca4b-427d-8f38-36079d5e4b95,DISK]] 2023-07-11 14:21:06,330 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:21:06,330 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-11 14:21:06,330 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-11 14:21:06,330 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-11 14:21:06,330 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-11 14:21:06,331 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:06,331 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-11 14:21:06,331 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-11 14:21:06,332 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-11 14:21:06,333 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/info 2023-07-11 14:21:06,333 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/info 2023-07-11 14:21:06,333 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-11 14:21:06,334 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:06,334 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-11 14:21:06,335 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/rep_barrier 2023-07-11 14:21:06,335 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/rep_barrier 2023-07-11 14:21:06,335 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-11 14:21:06,336 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:06,336 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-11 14:21:06,337 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/table 2023-07-11 14:21:06,337 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/table 2023-07-11 14:21:06,337 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-11 14:21:06,338 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:06,338 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740 2023-07-11 14:21:06,339 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740 2023-07-11 14:21:06,341 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-11 14:21:06,342 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-11 14:21:06,343 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61137941, jitterRate=-0.08897368609905243}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-11 14:21:06,343 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-11 14:21:06,343 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689085266308 2023-07-11 14:21:06,346 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-11 14:21:06,346 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-11 14:21:06,346 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,46501,1689085264192, state=OPEN 2023-07-11 14:21:06,350 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-11 14:21:06,350 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-11 14:21:06,350 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:21:06,350 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-11 14:21:06,351 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-11 14:21:06,351 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,46501,1689085264192 in 195 msec 2023-07-11 14:21:06,353 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-11 14:21:06,353 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 352 msec 2023-07-11 14:21:06,354 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1900 sec 2023-07-11 14:21:06,354 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689085266354, completionTime=-1 2023-07-11 14:21:06,354 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-11 14:21:06,354 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:21:06,357 DEBUG [hconnection-0x2bc53a79-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 14:21:06,358 INFO [RS-EventLoopGroup-17-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:33128, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 14:21:06,360 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:21:06,360 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085326360 2023-07-11 14:21:06,360 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085386360 2023-07-11 14:21:06,360 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-07-11 14:21:06,367 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,40067,1689085264137-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:06,367 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,40067,1689085264137-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:06,367 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,40067,1689085264137-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:06,367 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:40067, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:06,367 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:06,367 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-11 14:21:06,367 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-11 14:21:06,368 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-11 14:21:06,368 DEBUG [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-11 14:21:06,369 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-11 14:21:06,370 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-11 14:21:06,371 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/hbase/namespace/314c4626d8bf336ef8c434a723b560cf 2023-07-11 14:21:06,371 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/hbase/namespace/314c4626d8bf336ef8c434a723b560cf empty. 2023-07-11 14:21:06,372 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/hbase/namespace/314c4626d8bf336ef8c434a723b560cf 2023-07-11 14:21:06,372 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-11 14:21:06,780 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-11 14:21:06,781 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 314c4626d8bf336ef8c434a723b560cf, NAME => 'hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp 2023-07-11 14:21:07,187 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:07,187 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 314c4626d8bf336ef8c434a723b560cf, disabling compactions & flushes 2023-07-11 14:21:07,187 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf. 2023-07-11 14:21:07,187 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf. 2023-07-11 14:21:07,187 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf. after waiting 0 ms 2023-07-11 14:21:07,187 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf. 2023-07-11 14:21:07,187 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf. 2023-07-11 14:21:07,187 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 314c4626d8bf336ef8c434a723b560cf: 2023-07-11 14:21:07,190 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-11 14:21:07,190 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689085267190"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085267190"}]},"ts":"1689085267190"} 2023-07-11 14:21:07,193 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-11 14:21:07,193 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-11 14:21:07,194 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085267193"}]},"ts":"1689085267193"} 2023-07-11 14:21:07,195 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-11 14:21:07,197 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=314c4626d8bf336ef8c434a723b560cf, ASSIGN}] 2023-07-11 14:21:07,198 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=314c4626d8bf336ef8c434a723b560cf, ASSIGN 2023-07-11 14:21:07,199 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=314c4626d8bf336ef8c434a723b560cf, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,46501,1689085264192; forceNewPlan=false, retain=false 2023-07-11 14:21:07,350 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=314c4626d8bf336ef8c434a723b560cf, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:07,350 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689085267350"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085267350"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085267350"}]},"ts":"1689085267350"} 2023-07-11 14:21:07,352 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 314c4626d8bf336ef8c434a723b560cf, server=jenkins-hbase9.apache.org,46501,1689085264192}] 2023-07-11 14:21:07,507 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf. 2023-07-11 14:21:07,508 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 314c4626d8bf336ef8c434a723b560cf, NAME => 'hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:21:07,508 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 314c4626d8bf336ef8c434a723b560cf 2023-07-11 14:21:07,508 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:07,508 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 314c4626d8bf336ef8c434a723b560cf 2023-07-11 14:21:07,508 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 314c4626d8bf336ef8c434a723b560cf 2023-07-11 14:21:07,509 INFO [StoreOpener-314c4626d8bf336ef8c434a723b560cf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 314c4626d8bf336ef8c434a723b560cf 2023-07-11 14:21:07,511 DEBUG [StoreOpener-314c4626d8bf336ef8c434a723b560cf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/namespace/314c4626d8bf336ef8c434a723b560cf/info 2023-07-11 14:21:07,511 DEBUG [StoreOpener-314c4626d8bf336ef8c434a723b560cf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/namespace/314c4626d8bf336ef8c434a723b560cf/info 2023-07-11 14:21:07,511 INFO [StoreOpener-314c4626d8bf336ef8c434a723b560cf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 314c4626d8bf336ef8c434a723b560cf columnFamilyName info 2023-07-11 14:21:07,512 INFO [StoreOpener-314c4626d8bf336ef8c434a723b560cf-1] regionserver.HStore(310): Store=314c4626d8bf336ef8c434a723b560cf/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:07,512 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/namespace/314c4626d8bf336ef8c434a723b560cf 2023-07-11 14:21:07,513 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/namespace/314c4626d8bf336ef8c434a723b560cf 2023-07-11 14:21:07,516 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 314c4626d8bf336ef8c434a723b560cf 2023-07-11 14:21:07,518 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/namespace/314c4626d8bf336ef8c434a723b560cf/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:21:07,518 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 314c4626d8bf336ef8c434a723b560cf; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64383765, jitterRate=-0.040607139468193054}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 14:21:07,518 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 314c4626d8bf336ef8c434a723b560cf: 2023-07-11 14:21:07,519 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf., pid=6, masterSystemTime=1689085267504 2023-07-11 14:21:07,522 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf. 2023-07-11 14:21:07,522 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf. 2023-07-11 14:21:07,522 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=314c4626d8bf336ef8c434a723b560cf, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:07,522 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689085267522"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085267522"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085267522"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085267522"}]},"ts":"1689085267522"} 2023-07-11 14:21:07,525 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-11 14:21:07,525 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 314c4626d8bf336ef8c434a723b560cf, server=jenkins-hbase9.apache.org,46501,1689085264192 in 172 msec 2023-07-11 14:21:07,527 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-11 14:21:07,527 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=314c4626d8bf336ef8c434a723b560cf, ASSIGN in 328 msec 2023-07-11 14:21:07,528 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-11 14:21:07,528 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085267528"}]},"ts":"1689085267528"} 2023-07-11 14:21:07,529 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-11 14:21:07,531 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-11 14:21:07,532 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.1640 sec 2023-07-11 14:21:07,570 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-11 14:21:07,572 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-11 14:21:07,572 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:07,572 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:07,575 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-11 14:21:07,581 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-11 14:21:07,584 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 9 msec 2023-07-11 14:21:07,586 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:21:07,593 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-11 14:21:07,596 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 9 msec 2023-07-11 14:21:07,600 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-11 14:21:07,603 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-11 14:21:07,603 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.387sec 2023-07-11 14:21:07,603 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-11 14:21:07,603 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-11 14:21:07,603 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-11 14:21:07,603 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,40067,1689085264137-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-11 14:21:07,603 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,40067,1689085264137-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-11 14:21:07,605 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-11 14:21:07,610 DEBUG [Listener at localhost/45629] zookeeper.ReadOnlyZKClient(139): Connect 0x63e5fde6 to 127.0.0.1:55314 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:21:07,615 DEBUG [Listener at localhost/45629] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a034d59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:21:07,616 DEBUG [hconnection-0x1a5c5eee-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 14:21:07,617 INFO [RS-EventLoopGroup-17-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:33138, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 14:21:07,618 INFO [Listener at localhost/45629] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase9.apache.org,40067,1689085264137 2023-07-11 14:21:07,619 WARN [Listener at localhost/45629] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-11 14:21:07,621 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:07,622 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-11 14:21:07,622 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-11 14:21:07,626 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:07,626 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:07,626 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:07,626 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:07,626 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91 empty. 2023-07-11 14:21:07,626 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d empty. 2023-07-11 14:21:07,626 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b empty. 2023-07-11 14:21:07,626 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189 empty. 2023-07-11 14:21:07,627 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:07,627 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:07,627 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:07,627 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:07,627 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testEnableTableWithFailoverAtStep3 regions 2023-07-11 14:21:08,036 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp/data/default/testEnableTableWithFailoverAtStep3/.tabledesc/.tableinfo.0000000001 2023-07-11 14:21:08,038 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(7675): creating {ENCODED => ceb39529e4f4e8405895377233923b91, NAME => 'testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91.', STARTKEY => '', ENDKEY => 'a'}, tableDescriptor='testEnableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp 2023-07-11 14:21:08,038 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(7675): creating {ENCODED => e7a2de22dc894628b2b4c8217f03978b, NAME => 'testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b.', STARTKEY => 'a', ENDKEY => 'b'}, tableDescriptor='testEnableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp 2023-07-11 14:21:08,038 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(7675): creating {ENCODED => e7c7523fe03e3121494af249521c3c9d, NAME => 'testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d.', STARTKEY => 'b', ENDKEY => 'c'}, tableDescriptor='testEnableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp 2023-07-11 14:21:08,046 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:08,046 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1604): Closing e7a2de22dc894628b2b4c8217f03978b, disabling compactions & flushes 2023-07-11 14:21:08,046 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:08,046 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:08,046 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. after waiting 0 ms 2023-07-11 14:21:08,046 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:08,046 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:08,046 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1558): Region close journal for e7a2de22dc894628b2b4c8217f03978b: 2023-07-11 14:21:08,047 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(7675): creating {ENCODED => 6842bfd482b36498249bfd93acbcd189, NAME => 'testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189.', STARTKEY => 'c', ENDKEY => ''}, tableDescriptor='testEnableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.tmp 2023-07-11 14:21:08,445 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:08,445 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1604): Closing ceb39529e4f4e8405895377233923b91, disabling compactions & flushes 2023-07-11 14:21:08,445 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:08,445 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:08,445 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:08,445 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1604): Closing e7c7523fe03e3121494af249521c3c9d, disabling compactions & flushes 2023-07-11 14:21:08,445 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. after waiting 0 ms 2023-07-11 14:21:08,446 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:08,445 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:08,446 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:08,446 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:08,446 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. after waiting 0 ms 2023-07-11 14:21:08,446 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:08,446 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1558): Region close journal for ceb39529e4f4e8405895377233923b91: 2023-07-11 14:21:08,446 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:08,446 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1558): Region close journal for e7c7523fe03e3121494af249521c3c9d: 2023-07-11 14:21:08,451 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:08,451 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1604): Closing 6842bfd482b36498249bfd93acbcd189, disabling compactions & flushes 2023-07-11 14:21:08,451 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:08,451 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:08,451 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. after waiting 0 ms 2023-07-11 14:21:08,452 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:08,452 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:08,452 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1558): Region close journal for 6842bfd482b36498249bfd93acbcd189: 2023-07-11 14:21:08,454 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_ADD_TO_META 2023-07-11 14:21:08,455 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085268455"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085268455"}]},"ts":"1689085268455"} 2023-07-11 14:21:08,455 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689085268455"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085268455"}]},"ts":"1689085268455"} 2023-07-11 14:21:08,455 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085268455"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085268455"}]},"ts":"1689085268455"} 2023-07-11 14:21:08,455 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689085268455"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085268455"}]},"ts":"1689085268455"} 2023-07-11 14:21:08,459 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 4 regions to meta. 2023-07-11 14:21:08,460 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-11 14:21:08,460 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085268460"}]},"ts":"1689085268460"} 2023-07-11 14:21:08,461 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-11 14:21:08,464 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ceb39529e4f4e8405895377233923b91, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7a2de22dc894628b2b4c8217f03978b, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7c7523fe03e3121494af249521c3c9d, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=6842bfd482b36498249bfd93acbcd189, ASSIGN}] 2023-07-11 14:21:08,465 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=6842bfd482b36498249bfd93acbcd189, ASSIGN 2023-07-11 14:21:08,465 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=6842bfd482b36498249bfd93acbcd189, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,46501,1689085264192; forceNewPlan=false, retain=false 2023-07-11 14:21:08,466 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7c7523fe03e3121494af249521c3c9d, ASSIGN 2023-07-11 14:21:08,466 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7c7523fe03e3121494af249521c3c9d, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,46501,1689085264192; forceNewPlan=false, retain=false 2023-07-11 14:21:08,467 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7a2de22dc894628b2b4c8217f03978b, ASSIGN 2023-07-11 14:21:08,467 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7a2de22dc894628b2b4c8217f03978b, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,46501,1689085264192; forceNewPlan=false, retain=false 2023-07-11 14:21:08,467 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ceb39529e4f4e8405895377233923b91, ASSIGN 2023-07-11 14:21:08,468 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ceb39529e4f4e8405895377233923b91, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,46501,1689085264192; forceNewPlan=false, retain=false 2023-07-11 14:21:08,617 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=6842bfd482b36498249bfd93acbcd189, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:08,618 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689085268617"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085268617"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085268617"}]},"ts":"1689085268617"} 2023-07-11 14:21:08,620 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; OpenRegionProcedure 6842bfd482b36498249bfd93acbcd189, server=jenkins-hbase9.apache.org,46501,1689085264192}] 2023-07-11 14:21:08,622 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=e7c7523fe03e3121494af249521c3c9d, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:08,622 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085268622"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085268622"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085268622"}]},"ts":"1689085268622"} 2023-07-11 14:21:08,623 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE; OpenRegionProcedure e7c7523fe03e3121494af249521c3c9d, server=jenkins-hbase9.apache.org,46501,1689085264192}] 2023-07-11 14:21:08,625 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=e7a2de22dc894628b2b4c8217f03978b, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:08,625 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085268625"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085268625"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085268625"}]},"ts":"1689085268625"} 2023-07-11 14:21:08,626 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=11, state=RUNNABLE; OpenRegionProcedure e7a2de22dc894628b2b4c8217f03978b, server=jenkins-hbase9.apache.org,46501,1689085264192}] 2023-07-11 14:21:08,628 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=ceb39529e4f4e8405895377233923b91, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:08,628 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689085268628"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085268628"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085268628"}]},"ts":"1689085268628"} 2023-07-11 14:21:08,629 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=10, state=RUNNABLE; OpenRegionProcedure ceb39529e4f4e8405895377233923b91, server=jenkins-hbase9.apache.org,46501,1689085264192}] 2023-07-11 14:21:08,775 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:08,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ceb39529e4f4e8405895377233923b91, NAME => 'testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91.', STARTKEY => '', ENDKEY => 'a'} 2023-07-11 14:21:08,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:08,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:08,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:08,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:08,777 INFO [StoreOpener-ceb39529e4f4e8405895377233923b91-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:08,778 DEBUG [StoreOpener-ceb39529e4f4e8405895377233923b91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91/f1 2023-07-11 14:21:08,778 DEBUG [StoreOpener-ceb39529e4f4e8405895377233923b91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91/f1 2023-07-11 14:21:08,778 INFO [StoreOpener-ceb39529e4f4e8405895377233923b91-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ceb39529e4f4e8405895377233923b91 columnFamilyName f1 2023-07-11 14:21:08,779 INFO [StoreOpener-ceb39529e4f4e8405895377233923b91-1] regionserver.HStore(310): Store=ceb39529e4f4e8405895377233923b91/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:08,779 INFO [StoreOpener-ceb39529e4f4e8405895377233923b91-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:08,780 DEBUG [StoreOpener-ceb39529e4f4e8405895377233923b91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91/f2 2023-07-11 14:21:08,780 DEBUG [StoreOpener-ceb39529e4f4e8405895377233923b91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91/f2 2023-07-11 14:21:08,780 INFO [StoreOpener-ceb39529e4f4e8405895377233923b91-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ceb39529e4f4e8405895377233923b91 columnFamilyName f2 2023-07-11 14:21:08,781 INFO [StoreOpener-ceb39529e4f4e8405895377233923b91-1] regionserver.HStore(310): Store=ceb39529e4f4e8405895377233923b91/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:08,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:08,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:08,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:21:08,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:08,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:21:08,787 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened ceb39529e4f4e8405895377233923b91; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68330869, jitterRate=0.018209293484687805}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:21:08,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for ceb39529e4f4e8405895377233923b91: 2023-07-11 14:21:08,787 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91., pid=17, masterSystemTime=1689085268771 2023-07-11 14:21:08,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:08,789 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:08,789 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:08,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e7c7523fe03e3121494af249521c3c9d, NAME => 'testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d.', STARTKEY => 'b', ENDKEY => 'c'} 2023-07-11 14:21:08,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:08,789 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=ceb39529e4f4e8405895377233923b91, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:08,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:08,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:08,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:08,789 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689085268789"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085268789"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085268789"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085268789"}]},"ts":"1689085268789"} 2023-07-11 14:21:08,791 INFO [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:08,792 DEBUG [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d/f1 2023-07-11 14:21:08,792 DEBUG [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d/f1 2023-07-11 14:21:08,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=10 2023-07-11 14:21:08,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=10, state=SUCCESS; OpenRegionProcedure ceb39529e4f4e8405895377233923b91, server=jenkins-hbase9.apache.org,46501,1689085264192 in 162 msec 2023-07-11 14:21:08,792 INFO [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7c7523fe03e3121494af249521c3c9d columnFamilyName f1 2023-07-11 14:21:08,793 INFO [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] regionserver.HStore(310): Store=e7c7523fe03e3121494af249521c3c9d/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:08,793 INFO [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:08,794 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ceb39529e4f4e8405895377233923b91, ASSIGN in 329 msec 2023-07-11 14:21:08,795 DEBUG [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d/f2 2023-07-11 14:21:08,795 DEBUG [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d/f2 2023-07-11 14:21:08,795 INFO [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7c7523fe03e3121494af249521c3c9d columnFamilyName f2 2023-07-11 14:21:08,796 INFO [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] regionserver.HStore(310): Store=e7c7523fe03e3121494af249521c3c9d/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:08,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:08,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:08,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:21:08,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:08,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:21:08,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened e7c7523fe03e3121494af249521c3c9d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75020867, jitterRate=0.11789803206920624}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:21:08,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for e7c7523fe03e3121494af249521c3c9d: 2023-07-11 14:21:08,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d., pid=15, masterSystemTime=1689085268771 2023-07-11 14:21:08,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:08,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:08,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:08,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6842bfd482b36498249bfd93acbcd189, NAME => 'testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189.', STARTKEY => 'c', ENDKEY => ''} 2023-07-11 14:21:08,804 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=e7c7523fe03e3121494af249521c3c9d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:08,804 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085268804"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085268804"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085268804"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085268804"}]},"ts":"1689085268804"} 2023-07-11 14:21:08,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:08,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:08,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:08,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:08,806 INFO [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:08,807 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-11 14:21:08,807 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; OpenRegionProcedure e7c7523fe03e3121494af249521c3c9d, server=jenkins-hbase9.apache.org,46501,1689085264192 in 183 msec 2023-07-11 14:21:08,808 DEBUG [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189/f1 2023-07-11 14:21:08,808 DEBUG [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189/f1 2023-07-11 14:21:08,808 INFO [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6842bfd482b36498249bfd93acbcd189 columnFamilyName f1 2023-07-11 14:21:08,808 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7c7523fe03e3121494af249521c3c9d, ASSIGN in 343 msec 2023-07-11 14:21:08,809 INFO [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] regionserver.HStore(310): Store=6842bfd482b36498249bfd93acbcd189/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:08,809 INFO [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:08,810 DEBUG [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189/f2 2023-07-11 14:21:08,810 DEBUG [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189/f2 2023-07-11 14:21:08,810 INFO [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6842bfd482b36498249bfd93acbcd189 columnFamilyName f2 2023-07-11 14:21:08,811 INFO [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] regionserver.HStore(310): Store=6842bfd482b36498249bfd93acbcd189/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:08,811 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:08,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:08,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:21:08,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:08,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:21:08,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 6842bfd482b36498249bfd93acbcd189; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65887961, jitterRate=-0.018192872405052185}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:21:08,817 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 6842bfd482b36498249bfd93acbcd189: 2023-07-11 14:21:08,818 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189., pid=14, masterSystemTime=1689085268771 2023-07-11 14:21:08,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:08,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:08,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:08,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e7a2de22dc894628b2b4c8217f03978b, NAME => 'testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b.', STARTKEY => 'a', ENDKEY => 'b'} 2023-07-11 14:21:08,819 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=6842bfd482b36498249bfd93acbcd189, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:08,819 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689085268819"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085268819"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085268819"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085268819"}]},"ts":"1689085268819"} 2023-07-11 14:21:08,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:08,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:08,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:08,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:08,821 INFO [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:08,822 DEBUG [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b/f1 2023-07-11 14:21:08,822 DEBUG [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b/f1 2023-07-11 14:21:08,823 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-11 14:21:08,823 INFO [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7a2de22dc894628b2b4c8217f03978b columnFamilyName f1 2023-07-11 14:21:08,823 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; OpenRegionProcedure 6842bfd482b36498249bfd93acbcd189, server=jenkins-hbase9.apache.org,46501,1689085264192 in 201 msec 2023-07-11 14:21:08,823 INFO [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] regionserver.HStore(310): Store=e7a2de22dc894628b2b4c8217f03978b/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:08,823 INFO [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:08,824 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=6842bfd482b36498249bfd93acbcd189, ASSIGN in 359 msec 2023-07-11 14:21:08,825 DEBUG [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b/f2 2023-07-11 14:21:08,825 DEBUG [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b/f2 2023-07-11 14:21:08,825 INFO [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7a2de22dc894628b2b4c8217f03978b columnFamilyName f2 2023-07-11 14:21:08,826 INFO [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] regionserver.HStore(310): Store=e7a2de22dc894628b2b4c8217f03978b/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:08,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:08,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:08,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:21:08,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:08,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-11 14:21:08,832 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened e7a2de22dc894628b2b4c8217f03978b; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70799097, jitterRate=0.05498875677585602}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:21:08,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for e7a2de22dc894628b2b4c8217f03978b: 2023-07-11 14:21:08,832 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b., pid=16, masterSystemTime=1689085268771 2023-07-11 14:21:08,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:08,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:08,834 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=e7a2de22dc894628b2b4c8217f03978b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:08,834 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085268834"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085268834"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085268834"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085268834"}]},"ts":"1689085268834"} 2023-07-11 14:21:08,836 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=11 2023-07-11 14:21:08,836 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=11, state=SUCCESS; OpenRegionProcedure e7a2de22dc894628b2b4c8217f03978b, server=jenkins-hbase9.apache.org,46501,1689085264192 in 209 msec 2023-07-11 14:21:08,838 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=9 2023-07-11 14:21:08,838 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7a2de22dc894628b2b4c8217f03978b, ASSIGN in 372 msec 2023-07-11 14:21:08,838 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-11 14:21:08,838 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085268838"}]},"ts":"1689085268838"} 2023-07-11 14:21:08,839 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=ENABLED in hbase:meta 2023-07-11 14:21:08,841 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_POST_OPERATION 2023-07-11 14:21:08,842 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 in 1.2210 sec 2023-07-11 14:21:08,873 DEBUG [Listener at localhost/45629] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-11 14:21:08,875 INFO [RS-EventLoopGroup-15-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:41254, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-11 14:21:08,876 INFO [Listener at localhost/45629] client.HBaseAdmin$15(890): Started disable of testEnableTableWithFailoverAtStep3 2023-07-11 14:21:08,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40067] master.HMaster$11(2418): Client=jenkins//172.31.2.10 disable testEnableTableWithFailoverAtStep3 2023-07-11 14:21:08,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40067] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:08,879 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085268879"}]},"ts":"1689085268879"} 2023-07-11 14:21:08,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40067] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-11 14:21:08,880 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=DISABLING in hbase:meta 2023-07-11 14:21:08,883 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testEnableTableWithFailoverAtStep3 to state=DISABLING 2023-07-11 14:21:08,883 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ceb39529e4f4e8405895377233923b91, UNASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7a2de22dc894628b2b4c8217f03978b, UNASSIGN}, {pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7c7523fe03e3121494af249521c3c9d, UNASSIGN}, {pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=6842bfd482b36498249bfd93acbcd189, UNASSIGN}] 2023-07-11 14:21:08,884 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=6842bfd482b36498249bfd93acbcd189, UNASSIGN 2023-07-11 14:21:08,884 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=6842bfd482b36498249bfd93acbcd189, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:08,885 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689085268884"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085268884"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085268884"}]},"ts":"1689085268884"} 2023-07-11 14:21:08,886 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; CloseRegionProcedure 6842bfd482b36498249bfd93acbcd189, server=jenkins-hbase9.apache.org,46501,1689085264192}] 2023-07-11 14:21:08,887 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7c7523fe03e3121494af249521c3c9d, UNASSIGN 2023-07-11 14:21:08,888 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e7c7523fe03e3121494af249521c3c9d, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:08,888 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085268888"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085268888"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085268888"}]},"ts":"1689085268888"} 2023-07-11 14:21:08,889 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=21, state=RUNNABLE; CloseRegionProcedure e7c7523fe03e3121494af249521c3c9d, server=jenkins-hbase9.apache.org,46501,1689085264192}] 2023-07-11 14:21:08,890 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7a2de22dc894628b2b4c8217f03978b, UNASSIGN 2023-07-11 14:21:08,891 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=e7a2de22dc894628b2b4c8217f03978b, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:08,891 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085268891"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085268891"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085268891"}]},"ts":"1689085268891"} 2023-07-11 14:21:08,892 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=20, state=RUNNABLE; CloseRegionProcedure e7a2de22dc894628b2b4c8217f03978b, server=jenkins-hbase9.apache.org,46501,1689085264192}] 2023-07-11 14:21:08,894 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ceb39529e4f4e8405895377233923b91, UNASSIGN 2023-07-11 14:21:08,894 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=ceb39529e4f4e8405895377233923b91, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:08,894 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689085268894"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085268894"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085268894"}]},"ts":"1689085268894"} 2023-07-11 14:21:08,895 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=19, state=RUNNABLE; CloseRegionProcedure ceb39529e4f4e8405895377233923b91, server=jenkins-hbase9.apache.org,46501,1689085264192}] 2023-07-11 14:21:08,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40067] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-11 14:21:09,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(111): Close e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:09,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing e7a2de22dc894628b2b4c8217f03978b, disabling compactions & flushes 2023-07-11 14:21:09,038 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:09,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:09,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. after waiting 0 ms 2023-07-11 14:21:09,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:09,042 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-11 14:21:09,042 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:09,042 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for e7a2de22dc894628b2b4c8217f03978b: 2023-07-11 14:21:09,044 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(149): Closed e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:09,044 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(111): Close ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:09,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing ceb39529e4f4e8405895377233923b91, disabling compactions & flushes 2023-07-11 14:21:09,045 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:09,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:09,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. after waiting 0 ms 2023-07-11 14:21:09,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:09,045 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=e7a2de22dc894628b2b4c8217f03978b, regionState=CLOSED 2023-07-11 14:21:09,045 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085269045"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085269045"}]},"ts":"1689085269045"} 2023-07-11 14:21:09,048 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=20 2023-07-11 14:21:09,048 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=20, state=SUCCESS; CloseRegionProcedure e7a2de22dc894628b2b4c8217f03978b, server=jenkins-hbase9.apache.org,46501,1689085264192 in 154 msec 2023-07-11 14:21:09,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-11 14:21:09,049 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:09,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for ceb39529e4f4e8405895377233923b91: 2023-07-11 14:21:09,049 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7a2de22dc894628b2b4c8217f03978b, UNASSIGN in 165 msec 2023-07-11 14:21:09,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(149): Closed ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:09,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(111): Close 6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:09,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 6842bfd482b36498249bfd93acbcd189, disabling compactions & flushes 2023-07-11 14:21:09,051 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:09,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:09,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. after waiting 0 ms 2023-07-11 14:21:09,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:09,051 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=ceb39529e4f4e8405895377233923b91, regionState=CLOSED 2023-07-11 14:21:09,051 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689085269051"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085269051"}]},"ts":"1689085269051"} 2023-07-11 14:21:09,054 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=19 2023-07-11 14:21:09,054 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=19, state=SUCCESS; CloseRegionProcedure ceb39529e4f4e8405895377233923b91, server=jenkins-hbase9.apache.org,46501,1689085264192 in 157 msec 2023-07-11 14:21:09,054 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-11 14:21:09,054 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:09,054 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 6842bfd482b36498249bfd93acbcd189: 2023-07-11 14:21:09,055 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ceb39529e4f4e8405895377233923b91, UNASSIGN in 171 msec 2023-07-11 14:21:09,056 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(149): Closed 6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:09,056 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(111): Close e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:09,056 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing e7c7523fe03e3121494af249521c3c9d, disabling compactions & flushes 2023-07-11 14:21:09,056 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:09,056 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:09,056 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. after waiting 0 ms 2023-07-11 14:21:09,056 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=6842bfd482b36498249bfd93acbcd189, regionState=CLOSED 2023-07-11 14:21:09,056 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:09,056 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689085269056"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085269056"}]},"ts":"1689085269056"} 2023-07-11 14:21:09,059 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-11 14:21:09,059 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; CloseRegionProcedure 6842bfd482b36498249bfd93acbcd189, server=jenkins-hbase9.apache.org,46501,1689085264192 in 172 msec 2023-07-11 14:21:09,059 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-11 14:21:09,060 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:09,060 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for e7c7523fe03e3121494af249521c3c9d: 2023-07-11 14:21:09,060 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=6842bfd482b36498249bfd93acbcd189, UNASSIGN in 176 msec 2023-07-11 14:21:09,061 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(149): Closed e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:09,061 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e7c7523fe03e3121494af249521c3c9d, regionState=CLOSED 2023-07-11 14:21:09,061 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085269061"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689085269061"}]},"ts":"1689085269061"} 2023-07-11 14:21:09,064 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=21 2023-07-11 14:21:09,064 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=21, state=SUCCESS; CloseRegionProcedure e7c7523fe03e3121494af249521c3c9d, server=jenkins-hbase9.apache.org,46501,1689085264192 in 174 msec 2023-07-11 14:21:09,065 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=18 2023-07-11 14:21:09,065 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7c7523fe03e3121494af249521c3c9d, UNASSIGN in 181 msec 2023-07-11 14:21:09,065 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085269065"}]},"ts":"1689085269065"} 2023-07-11 14:21:09,066 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=DISABLED in hbase:meta 2023-07-11 14:21:09,068 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testEnableTableWithFailoverAtStep3 to state=DISABLED 2023-07-11 14:21:09,069 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 in 192 msec 2023-07-11 14:21:09,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40067] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-11 14:21:09,182 INFO [Listener at localhost/45629] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testEnableTableWithFailoverAtStep3, procId: 18 completed 2023-07-11 14:21:09,182 WARN [Listener at localhost/45629] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-07-11 14:21:09,188 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,190 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:21:09,190 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,190 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:21:09,190 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:21:09,438 INFO [Listener at localhost/45629] procedure.TestMasterFailoverWithProcedures(306): Restart 0 exec state: pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,438 INFO [Listener at localhost/45629] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:21:09,438 INFO [Listener at localhost/45629] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:21:09,438 INFO [Listener at localhost/45629] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:21:09,440 ERROR [Listener at localhost/45629] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:21:09,440 INFO [Listener at localhost/45629] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:21:09,440 INFO [Listener at localhost/45629] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:21:09,440 INFO [Listener at localhost/45629] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:21:09,440 INFO [Listener at localhost/45629] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:21:09,441 INFO [Listener at localhost/45629] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:21:09,442 DEBUG [Listener at localhost/45629] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.EnableTableProcedure) stack ids=[] 2023-07-11 14:21:09,442 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:21:09,442 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:21:09,443 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:21:09,443 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:21:09,443 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,443 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,443 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,443 DEBUG [Listener at localhost/45629] procedure2.Procedure(966): pid=27, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:21:09,444 INFO [Listener at localhost/45629] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-11 14:21:09,444 DEBUG [Listener at localhost/45629] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:21:09,444 DEBUG [Listener at localhost/45629] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,46501,1689085264192, table=hbase:meta, region=1588230740 2023-07-11 14:21:09,445 INFO [Listener at localhost/45629] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085299445 2023-07-11 14:21:09,445 DEBUG [Listener at localhost/45629] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:21:09,446 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,448 DEBUG [Listener at localhost/45629] assignment.RegionStateStore(147): Load hbase:meta entry region=314c4626d8bf336ef8c434a723b560cf, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:09,448 DEBUG [Listener at localhost/45629] assignment.RegionStateStore(147): Load hbase:meta entry region=ceb39529e4f4e8405895377233923b91, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:09,448 DEBUG [Listener at localhost/45629] assignment.RegionStateStore(147): Load hbase:meta entry region=e7a2de22dc894628b2b4c8217f03978b, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:09,448 DEBUG [Listener at localhost/45629] assignment.RegionStateStore(147): Load hbase:meta entry region=e7c7523fe03e3121494af249521c3c9d, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:09,448 DEBUG [Listener at localhost/45629] assignment.RegionStateStore(147): Load hbase:meta entry region=6842bfd482b36498249bfd93acbcd189, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:09,448 INFO [Listener at localhost/45629] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:21:09,448 INFO [Listener at localhost/45629] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085329448 2023-07-11 14:21:09,449 INFO [Listener at localhost/45629] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085389449 2023-07-11 14:21:09,449 INFO [Listener at localhost/45629] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-11 14:21:09,449 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:21:09,450 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:21:09,450 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_SET_ENABLING_TABLE_STATE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,450 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:21:09,450 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_SET_ENABLING_TABLE_STATE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:21:09,699 INFO [Listener at localhost/45629] procedure.TestMasterFailoverWithProcedures(306): Restart 1 exec state: pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,699 INFO [Listener at localhost/45629] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:21:09,699 INFO [Listener at localhost/45629] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:21:09,699 INFO [Listener at localhost/45629] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:21:09,699 ERROR [Listener at localhost/45629] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:21:09,699 INFO [Listener at localhost/45629] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:21:09,699 INFO [Listener at localhost/45629] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:21:09,699 INFO [Listener at localhost/45629] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:21:09,700 INFO [Listener at localhost/45629] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:21:09,700 INFO [Listener at localhost/45629] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:21:09,701 DEBUG [Listener at localhost/45629] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.EnableTableProcedure) stack ids=[0] 2023-07-11 14:21:09,701 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:21:09,702 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:21:09,702 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:21:09,702 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:21:09,702 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,703 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,703 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,703 DEBUG [Listener at localhost/45629] procedure2.Procedure(966): pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:21:09,703 INFO [Listener at localhost/45629] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-11 14:21:09,704 DEBUG [Listener at localhost/45629] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:21:09,704 DEBUG [Listener at localhost/45629] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,46501,1689085264192, table=hbase:meta, region=1588230740 2023-07-11 14:21:09,705 INFO [Listener at localhost/45629] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085299705 2023-07-11 14:21:09,708 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,708 DEBUG [Listener at localhost/45629] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:21:09,710 DEBUG [Listener at localhost/45629] assignment.RegionStateStore(147): Load hbase:meta entry region=314c4626d8bf336ef8c434a723b560cf, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:09,710 DEBUG [Listener at localhost/45629] assignment.RegionStateStore(147): Load hbase:meta entry region=ceb39529e4f4e8405895377233923b91, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:09,710 DEBUG [Listener at localhost/45629] assignment.RegionStateStore(147): Load hbase:meta entry region=e7a2de22dc894628b2b4c8217f03978b, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:09,710 DEBUG [Listener at localhost/45629] assignment.RegionStateStore(147): Load hbase:meta entry region=e7c7523fe03e3121494af249521c3c9d, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:09,711 DEBUG [Listener at localhost/45629] assignment.RegionStateStore(147): Load hbase:meta entry region=6842bfd482b36498249bfd93acbcd189, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:09,711 INFO [Listener at localhost/45629] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:21:09,711 INFO [Listener at localhost/45629] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085329711 2023-07-11 14:21:09,711 INFO [Listener at localhost/45629] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085389711 2023-07-11 14:21:09,711 INFO [Listener at localhost/45629] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-11 14:21:09,711 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:21:09,713 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,713 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085269713"}]},"ts":"1689085269713"} 2023-07-11 14:21:09,714 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-11 14:21:09,715 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:21:09,715 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_MARK_REGIONS_ONLINE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,715 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:21:09,716 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_MARK_REGIONS_ONLINE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:21:09,961 INFO [Listener at localhost/45629] procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state: pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,961 INFO [Listener at localhost/45629] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-11 14:21:09,961 INFO [Listener at localhost/45629] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:21:09,961 INFO [Listener at localhost/45629] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:21:09,962 ERROR [Listener at localhost/45629] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:21:09,962 INFO [Listener at localhost/45629] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-11 14:21:09,962 INFO [Listener at localhost/45629] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:21:09,963 INFO [Listener at localhost/45629] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:21:09,963 INFO [Listener at localhost/45629] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:21:09,963 INFO [Listener at localhost/45629] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:21:09,964 DEBUG [Listener at localhost/45629] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.EnableTableProcedure) stack ids=[0, 1] 2023-07-11 14:21:09,964 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:21:09,965 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:21:09,965 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:21:09,965 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:21:09,965 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,965 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,966 DEBUG [Listener at localhost/45629] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:ENABLE_TABLE_SET_ENABLING_TABLE_STATE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,966 DEBUG [Listener at localhost/45629] procedure2.Procedure(966): pid=27, state=RUNNABLE:ENABLE_TABLE_SET_ENABLING_TABLE_STATE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:21:09,966 INFO [Listener at localhost/45629] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-11 14:21:09,966 DEBUG [Listener at localhost/45629] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:21:09,967 DEBUG [Listener at localhost/45629] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,46501,1689085264192, table=hbase:meta, region=1588230740 2023-07-11 14:21:09,967 INFO [Listener at localhost/45629] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085299967 2023-07-11 14:21:09,968 DEBUG [Listener at localhost/45629] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:21:09,968 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:ENABLE_TABLE_SET_ENABLING_TABLE_STATE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,971 DEBUG [Listener at localhost/45629] assignment.RegionStateStore(147): Load hbase:meta entry region=314c4626d8bf336ef8c434a723b560cf, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:09,971 DEBUG [Listener at localhost/45629] assignment.RegionStateStore(147): Load hbase:meta entry region=ceb39529e4f4e8405895377233923b91, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:09,971 DEBUG [Listener at localhost/45629] assignment.RegionStateStore(147): Load hbase:meta entry region=e7a2de22dc894628b2b4c8217f03978b, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:09,971 DEBUG [Listener at localhost/45629] assignment.RegionStateStore(147): Load hbase:meta entry region=e7c7523fe03e3121494af249521c3c9d, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:09,971 DEBUG [Listener at localhost/45629] assignment.RegionStateStore(147): Load hbase:meta entry region=6842bfd482b36498249bfd93acbcd189, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:09,971 INFO [Listener at localhost/45629] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:21:09,971 INFO [Listener at localhost/45629] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085329971 2023-07-11 14:21:09,972 INFO [Listener at localhost/45629] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085389972 2023-07-11 14:21:09,972 INFO [Listener at localhost/45629] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-07-11 14:21:09,972 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,972 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085269972"}]},"ts":"1689085269972"} 2023-07-11 14:21:09,973 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-11 14:21:09,975 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-11 14:21:09,977 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-07-11 14:21:09,978 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ceb39529e4f4e8405895377233923b91, ASSIGN}, {pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7a2de22dc894628b2b4c8217f03978b, ASSIGN}, {pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7c7523fe03e3121494af249521c3c9d, ASSIGN}, {pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=6842bfd482b36498249bfd93acbcd189, ASSIGN}] 2023-07-11 14:21:09,978 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-11 14:21:09,978 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=WAITING:ENABLE_TABLE_SET_ENABLED_TABLE_STATE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:09,978 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:21:09,978 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=WAITING:ENABLE_TABLE_SET_ENABLED_TABLE_STATE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-11 14:21:10,021 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-11 14:21:10,222 INFO [Listener at localhost/45629] procedure.TestMasterFailoverWithProcedures(313): Trigger master failover 2023-07-11 14:21:10,222 INFO [Listener at localhost/45629] hbase.MiniHBaseCluster(607): Aborting jenkins-hbase9.apache.org,40067,1689085264137 2023-07-11 14:21:10,222 ERROR [Listener at localhost/45629] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-11 14:21:10,222 ERROR [Listener at localhost/45629] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase9.apache.org,40067,1689085264137: Aborting for tests ***** java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:608) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:411) at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.masterFailover(MasterProcedureTestingUtility.java:135) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testRecoveryAndDoubleExecution(TestMasterFailoverWithProcedures.java:314) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testEnableTableWithFailoverAtStep(TestMasterFailoverWithProcedures.java:289) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testEnableTableWithFailover(TestMasterFailoverWithProcedures.java:270) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-11 14:21:10,222 INFO [Listener at localhost/45629] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,40067,1689085264137' ***** 2023-07-11 14:21:10,222 INFO [Listener at localhost/45629] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost/45629 2023-07-11 14:21:10,223 DEBUG [M:0;jenkins-hbase9:40067] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@550f74b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-11 14:21:10,223 INFO [M:0;jenkins-hbase9:40067] regionserver.HRegionServer(1141): aborting server jenkins-hbase9.apache.org,40067,1689085264137 2023-07-11 14:21:10,224 DEBUG [M:0;jenkins-hbase9:40067] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4aec6543 to 127.0.0.1:55314 2023-07-11 14:21:10,224 DEBUG [M:0;jenkins-hbase9:40067] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:21:10,224 INFO [M:0;jenkins-hbase9:40067] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,40067,1689085264137; all regions closed. 2023-07-11 14:21:10,224 DEBUG [M:0;jenkins-hbase9:40067] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:21:10,224 DEBUG [M:0;jenkins-hbase9:40067] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-11 14:21:10,224 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-11 14:21:10,224 DEBUG [M:0;jenkins-hbase9:40067] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-11 14:21:10,224 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085265168] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085265168,5,FailOnTimeoutGroup] 2023-07-11 14:21:10,224 INFO [M:0;jenkins-hbase9:40067] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-11 14:21:10,225 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085265168] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085265168,5,FailOnTimeoutGroup] 2023-07-11 14:21:10,225 INFO [M:0;jenkins-hbase9:40067] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-11 14:21:10,226 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:21:10,226 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:21:10,226 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:10,226 INFO [M:0;jenkins-hbase9:40067] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-07-11 14:21:10,226 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:10,226 DEBUG [M:0;jenkins-hbase9:40067] master.HMaster(1512): Stopping service threads 2023-07-11 14:21:10,226 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): regionserver:46501-0x10154b4830b0002, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:21:10,226 INFO [M:0;jenkins-hbase9:40067] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-11 14:21:10,227 ERROR [M:0;jenkins-hbase9:40067] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:21:10,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:21:10,227 INFO [M:0;jenkins-hbase9:40067] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-11 14:21:10,227 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-11 14:21:10,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:21:10,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:21:10,228 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-11 14:21:10,228 DEBUG [M:0;jenkins-hbase9:40067] zookeeper.ZKUtil(398): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-11 14:21:10,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46501-0x10154b4830b0002, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:21:10,228 WARN [M:0;jenkins-hbase9:40067] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-11 14:21:10,228 INFO [M:0;jenkins-hbase9:40067] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:21:10,228 INFO [M:0;jenkins-hbase9:40067] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-11 14:21:10,229 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): regionserver:46501-0x10154b4830b0002, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:21:10,230 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:21:10,230 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:10,230 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-11 14:21:10,230 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:10,231 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:21:10,231 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,40895,1689085264170 from backup master directory 2023-07-11 14:21:10,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-11 14:21:10,232 DEBUG [M:0;jenkins-hbase9:40067] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 14:21:10,232 INFO [M:0;jenkins-hbase9:40067] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:10,232 DEBUG [M:0;jenkins-hbase9:40067] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:10,233 DEBUG [M:0;jenkins-hbase9:40067] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 14:21:10,233 DEBUG [M:0;jenkins-hbase9:40067] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:10,233 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:21:10,233 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,40895,1689085264170 2023-07-11 14:21:10,233 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-11 14:21:10,233 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-11 14:21:10,233 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase9.apache.org,40895,1689085264170 2023-07-11 14:21:10,234 ERROR [M:0;jenkins-hbase9:40067] regionserver.HRegion(1825): Memstore data size is 89379 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:10,234 INFO [M:0;jenkins-hbase9:40067] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:10,234 DEBUG [M:0;jenkins-hbase9:40067] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:21:10,234 INFO [M:0;jenkins-hbase9:40067] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-11 14:21:10,234 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 14:21:10,234 INFO [M:0;jenkins-hbase9:40067] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:40067 2023-07-11 14:21:10,237 DEBUG [M:0;jenkins-hbase9:40067] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,40067,1689085264137 already deleted, retry=false 2023-07-11 14:21:10,238 WARN [RS-EventLoopGroup-17-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase9.apache.org/172.31.2.10:40067 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:40067 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-11 14:21:10,238 DEBUG [RS-EventLoopGroup-17-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase9.apache.org/172.31.2.10:40067 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase9.apache.org/172.31.2.10:40067 2023-07-11 14:21:10,247 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-11 14:21:10,268 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x11642b7f to 127.0.0.1:55314 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-11 14:21:10,272 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ebd358d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-11 14:21:10,273 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-11 14:21:10,273 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-11 14:21:10,273 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-11 14:21:10,280 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/WALs/jenkins-hbase9.apache.org,40067,1689085264137 to hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/WALs/jenkins-hbase9.apache.org,40067,1689085264137-dead as it is dead 2023-07-11 14:21:10,280 INFO [master/jenkins-hbase9:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/WALs/jenkins-hbase9.apache.org,40067,1689085264137-dead/jenkins-hbase9.apache.org%2C40067%2C1689085264137.1689085265099 2023-07-11 14:21:10,280 INFO [master/jenkins-hbase9:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/WALs/jenkins-hbase9.apache.org,40067,1689085264137-dead/jenkins-hbase9.apache.org%2C40067%2C1689085264137.1689085265099 after 0ms 2023-07-11 14:21:10,281 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/WALs/jenkins-hbase9.apache.org,40067,1689085264137-dead/jenkins-hbase9.apache.org%2C40067%2C1689085264137.1689085265099 to hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C40067%2C1689085264137.1689085265099 2023-07-11 14:21:10,281 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/WALs/jenkins-hbase9.apache.org,40067,1689085264137-dead 2023-07-11 14:21:10,281 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/WALs/jenkins-hbase9.apache.org,40895,1689085264170 2023-07-11 14:21:10,283 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C40895%2C1689085264170, suffix=, logDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/WALs/jenkins-hbase9.apache.org,40895,1689085264170, archiveDir=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/oldWALs, maxLogs=10 2023-07-11 14:21:10,298 DEBUG [RS-EventLoopGroup-17-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45335,DS-b87c366b-ca4b-427d-8f38-36079d5e4b95,DISK] 2023-07-11 14:21:10,300 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/WALs/jenkins-hbase9.apache.org,40895,1689085264170/jenkins-hbase9.apache.org%2C40895%2C1689085264170.1689085270283 2023-07-11 14:21:10,300 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45335,DS-b87c366b-ca4b-427d-8f38-36079d5e4b95,DISK]] 2023-07-11 14:21:10,300 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-11 14:21:10,300 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:10,300 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:21:10,301 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:21:10,302 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:21:10,303 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-11 14:21:10,304 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-11 14:21:10,304 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:10,305 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-11 14:21:10,305 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C40067%2C1689085264137.1689085265099 2023-07-11 14:21:10,312 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 231, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=193, path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C40067%2C1689085264137.1689085265099 2023-07-11 14:21:10,312 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=87.28 KB heapSize=104.57 KB 2023-07-11 14:21:10,322 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=87.28 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/4459def924754a198c752626a9adf19d 2023-07-11 14:21:10,329 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/4459def924754a198c752626a9adf19d as hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/4459def924754a198c752626a9adf19d 2023-07-11 14:21:10,334 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/4459def924754a198c752626a9adf19d, entries=231, sequenceid=193, filesize=92.5 K 2023-07-11 14:21:10,334 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~87.28 KB/89379, heapSize ~104.55 KB/107064, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 22ms, sequenceid=193, compaction requested=false; wal=null 2023-07-11 14:21:10,335 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase9.apache.org%2C40067%2C1689085264137.1689085265099 2023-07-11 14:21:10,338 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-11 14:21:10,340 DEBUG [RS:0;jenkins-hbase9:46501] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-11 14:21:10,340 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/193.seqid, newMaxSeqId=193, maxSeqId=1 2023-07-11 14:21:10,342 INFO [RS-EventLoopGroup-16-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:51533, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-07-11 14:21:10,342 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=194; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72147322, jitterRate=0.07507887482643127}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-11 14:21:10,342 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:21:10,342 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-11 14:21:10,342 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40895] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-11 14:21:10,343 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-11 14:21:10,343 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-11 14:21:10,344 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-11 14:21:10,344 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-11 14:21:10,348 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.EnableTableProcedure) stack ids=[0, 1, 2] 2023-07-11 14:21:10,348 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-11 14:21:10,348 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-11 14:21:10,349 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-11 14:21:10,349 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-11 14:21:10,349 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:10,349 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:10,349 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:ENABLE_TABLE_MARK_REGIONS_ONLINE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:10,349 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.Procedure(966): pid=27, state=RUNNABLE:ENABLE_TABLE_MARK_REGIONS_ONLINE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-11 14:21:10,349 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-07-11 14:21:10,349 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-11 14:21:10,353 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-11 14:21:10,353 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,46501,1689085264192, table=hbase:meta, region=1588230740 2023-07-11 14:21:10,355 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 1 possibly 'live' servers, and 0 'splitting'. 2023-07-11 14:21:10,355 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:10,368 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-11 14:21:10,368 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-11 14:21:10,369 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-11 14:21:10,369 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-11 14:21:10,370 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-11 14:21:10,371 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-11 14:21:10,371 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-11 14:21:10,372 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-11 14:21:10,372 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase9.apache.org,40895,1689085264170, sessionid=0x10154b4830b0001, setting cluster-up flag (Was=true) 2023-07-11 14:21:10,374 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-11 14:21:10,377 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,40895,1689085264170 2023-07-11 14:21:10,380 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-11 14:21:10,381 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,40895,1689085264170 2023-07-11 14:21:10,382 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/.hbase-snapshot/.tmp 2023-07-11 14:21:10,383 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:21:10,383 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:21:10,383 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:21:10,383 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-07-11 14:21:10,383 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-07-11 14:21:10,383 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:21:10,383 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-07-11 14:21:10,383 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-07-11 14:21:10,384 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689085300384 2023-07-11 14:21:10,385 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:ENABLE_TABLE_MARK_REGIONS_ONLINE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-11 14:21:10,385 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-11 14:21:10,385 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-11 14:21:10,385 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-11 14:21:10,385 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-11 14:21:10,386 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-11 14:21:10,386 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-11 14:21:10,387 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:10,387 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-11 14:21:10,387 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-11 14:21:10,387 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-11 14:21:10,388 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-11 14:21:10,388 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-11 14:21:10,388 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085270388,5,FailOnTimeoutGroup] 2023-07-11 14:21:10,388 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085270388,5,FailOnTimeoutGroup] 2023-07-11 14:21:10,388 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:10,388 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-11 14:21:10,388 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:10,388 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:10,388 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689085270388, completionTime=-1 2023-07-11 14:21:10,388 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-11 14:21:10,388 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-11 14:21:10,390 DEBUG [hconnection-0x2483f811-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-11 14:21:10,391 INFO [RS-EventLoopGroup-17-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:33152, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-11 14:21:10,393 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=314c4626d8bf336ef8c434a723b560cf, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:10,393 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=ceb39529e4f4e8405895377233923b91, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:10,393 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=e7a2de22dc894628b2b4c8217f03978b, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:10,394 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=e7c7523fe03e3121494af249521c3c9d, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:10,394 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=6842bfd482b36498249bfd93acbcd189, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,46501,1689085264192, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192, openSeqNum=2 2023-07-11 14:21:10,394 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-11 14:21:10,394 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689085330394 2023-07-11 14:21:10,394 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689085390394 2023-07-11 14:21:10,394 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-07-11 14:21:10,405 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,40895,1689085264170-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:10,405 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,40895,1689085264170-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:10,405 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,40895,1689085264170-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:10,405 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:40895, period=300000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:10,405 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-11 14:21:10,405 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-11 14:21:10,407 DEBUG [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-11 14:21:10,413 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-11 14:21:10,418 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-11 14:21:10,418 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 0.185sec 2023-07-11 14:21:10,418 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-11 14:21:10,418 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-11 14:21:10,418 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-07-11 14:21:10,418 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-11 14:21:10,418 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,40895,1689085264170-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-11 14:21:10,418 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,40895,1689085264170-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-11 14:21:10,418 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ceb39529e4f4e8405895377233923b91, ASSIGN}, {pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7a2de22dc894628b2b4c8217f03978b, ASSIGN}, {pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7c7523fe03e3121494af249521c3c9d, ASSIGN}, {pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=6842bfd482b36498249bfd93acbcd189, ASSIGN}] 2023-07-11 14:21:10,422 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=6842bfd482b36498249bfd93acbcd189, ASSIGN 2023-07-11 14:21:10,422 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-11 14:21:10,422 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:10,422 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=6842bfd482b36498249bfd93acbcd189, ASSIGN; state=CLOSED, location=jenkins-hbase9.apache.org,46501,1689085264192; forceNewPlan=false, retain=true 2023-07-11 14:21:10,423 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7c7523fe03e3121494af249521c3c9d, ASSIGN 2023-07-11 14:21:10,423 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:10,423 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7c7523fe03e3121494af249521c3c9d, ASSIGN; state=CLOSED, location=jenkins-hbase9.apache.org,46501,1689085264192; forceNewPlan=false, retain=true 2023-07-11 14:21:10,424 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7a2de22dc894628b2b4c8217f03978b, ASSIGN 2023-07-11 14:21:10,424 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:10,424 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7a2de22dc894628b2b4c8217f03978b, ASSIGN; state=CLOSED, location=jenkins-hbase9.apache.org,46501,1689085264192; forceNewPlan=false, retain=true 2023-07-11 14:21:10,425 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ceb39529e4f4e8405895377233923b91, ASSIGN 2023-07-11 14:21:10,425 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:10,425 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ceb39529e4f4e8405895377233923b91, ASSIGN; state=CLOSED, location=jenkins-hbase9.apache.org,46501,1689085264192; forceNewPlan=false, retain=true 2023-07-11 14:21:10,427 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:21:10,427 INFO [M:0;jenkins-hbase9:40067] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,40067,1689085264137; zookeeper connection closed. 2023-07-11 14:21:10,427 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40067-0x10154b4830b0000, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:21:10,575 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=6842bfd482b36498249bfd93acbcd189, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:10,576 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689085270575"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085270575"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085270575"}]},"ts":"1689085270575"} 2023-07-11 14:21:10,578 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE; OpenRegionProcedure 6842bfd482b36498249bfd93acbcd189, server=jenkins-hbase9.apache.org,46501,1689085264192}] 2023-07-11 14:21:10,581 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=e7c7523fe03e3121494af249521c3c9d, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:10,581 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085270581"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085270581"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085270581"}]},"ts":"1689085270581"} 2023-07-11 14:21:10,582 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=30, state=RUNNABLE; OpenRegionProcedure e7c7523fe03e3121494af249521c3c9d, server=jenkins-hbase9.apache.org,46501,1689085264192}] 2023-07-11 14:21:10,584 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=e7a2de22dc894628b2b4c8217f03978b, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:10,585 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085270584"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085270584"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085270584"}]},"ts":"1689085270584"} 2023-07-11 14:21:10,586 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=29, state=RUNNABLE; OpenRegionProcedure e7a2de22dc894628b2b4c8217f03978b, server=jenkins-hbase9.apache.org,46501,1689085264192}] 2023-07-11 14:21:10,588 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=ceb39529e4f4e8405895377233923b91, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:10,588 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689085270588"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689085270588"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689085270588"}]},"ts":"1689085270588"} 2023-07-11 14:21:10,590 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=35, ppid=28, state=RUNNABLE; OpenRegionProcedure ceb39529e4f4e8405895377233923b91, server=jenkins-hbase9.apache.org,46501,1689085264192}] 2023-07-11 14:21:10,731 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:10,731 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-11 14:21:10,732 INFO [RS-EventLoopGroup-17-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:33156, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-11 14:21:10,735 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:10,735 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e7a2de22dc894628b2b4c8217f03978b, NAME => 'testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b.', STARTKEY => 'a', ENDKEY => 'b'} 2023-07-11 14:21:10,736 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:10,736 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:10,736 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:10,736 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:10,737 INFO [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:10,738 DEBUG [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b/f1 2023-07-11 14:21:10,738 DEBUG [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b/f1 2023-07-11 14:21:10,738 INFO [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7a2de22dc894628b2b4c8217f03978b columnFamilyName f1 2023-07-11 14:21:10,739 INFO [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] regionserver.HStore(310): Store=e7a2de22dc894628b2b4c8217f03978b/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:10,739 INFO [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:10,739 DEBUG [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b/f2 2023-07-11 14:21:10,739 DEBUG [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b/f2 2023-07-11 14:21:10,740 INFO [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7a2de22dc894628b2b4c8217f03978b columnFamilyName f2 2023-07-11 14:21:10,740 INFO [StoreOpener-e7a2de22dc894628b2b4c8217f03978b-1] regionserver.HStore(310): Store=e7a2de22dc894628b2b4c8217f03978b/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:10,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:10,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:10,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:21:10,744 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:10,745 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened e7a2de22dc894628b2b4c8217f03978b; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73871850, jitterRate=0.10077634453773499}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:21:10,745 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for e7a2de22dc894628b2b4c8217f03978b: 2023-07-11 14:21:10,745 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b., pid=34, masterSystemTime=1689085270731 2023-07-11 14:21:10,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:10,747 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:10,747 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:10,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6842bfd482b36498249bfd93acbcd189, NAME => 'testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189.', STARTKEY => 'c', ENDKEY => ''} 2023-07-11 14:21:10,747 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=e7a2de22dc894628b2b4c8217f03978b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:10,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:10,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:10,748 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085270747"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085270747"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085270747"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085270747"}]},"ts":"1689085270747"} 2023-07-11 14:21:10,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:10,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:10,749 INFO [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:10,750 DEBUG [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189/f1 2023-07-11 14:21:10,750 DEBUG [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189/f1 2023-07-11 14:21:10,750 INFO [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6842bfd482b36498249bfd93acbcd189 columnFamilyName f1 2023-07-11 14:21:10,750 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=29 2023-07-11 14:21:10,750 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=29, state=SUCCESS; OpenRegionProcedure e7a2de22dc894628b2b4c8217f03978b, server=jenkins-hbase9.apache.org,46501,1689085264192 in 163 msec 2023-07-11 14:21:10,751 INFO [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] regionserver.HStore(310): Store=6842bfd482b36498249bfd93acbcd189/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:10,751 INFO [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:10,751 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7a2de22dc894628b2b4c8217f03978b, ASSIGN in 333 msec 2023-07-11 14:21:10,751 DEBUG [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189/f2 2023-07-11 14:21:10,752 DEBUG [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189/f2 2023-07-11 14:21:10,752 INFO [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6842bfd482b36498249bfd93acbcd189 columnFamilyName f2 2023-07-11 14:21:10,752 INFO [StoreOpener-6842bfd482b36498249bfd93acbcd189-1] regionserver.HStore(310): Store=6842bfd482b36498249bfd93acbcd189/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:10,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:10,754 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:10,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:21:10,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:10,757 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 6842bfd482b36498249bfd93acbcd189; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63569694, jitterRate=-0.05273774266242981}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:21:10,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 6842bfd482b36498249bfd93acbcd189: 2023-07-11 14:21:10,758 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189., pid=32, masterSystemTime=1689085270731 2023-07-11 14:21:10,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:10,759 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:10,759 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:10,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e7c7523fe03e3121494af249521c3c9d, NAME => 'testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d.', STARTKEY => 'b', ENDKEY => 'c'} 2023-07-11 14:21:10,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:10,759 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=6842bfd482b36498249bfd93acbcd189, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:10,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:10,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:10,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:10,760 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689085270759"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085270759"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085270759"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085270759"}]},"ts":"1689085270759"} 2023-07-11 14:21:10,761 INFO [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:10,762 DEBUG [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d/f1 2023-07-11 14:21:10,762 DEBUG [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d/f1 2023-07-11 14:21:10,762 INFO [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7c7523fe03e3121494af249521c3c9d columnFamilyName f1 2023-07-11 14:21:10,762 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-07-11 14:21:10,762 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; OpenRegionProcedure 6842bfd482b36498249bfd93acbcd189, server=jenkins-hbase9.apache.org,46501,1689085264192 in 183 msec 2023-07-11 14:21:10,762 INFO [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] regionserver.HStore(310): Store=e7c7523fe03e3121494af249521c3c9d/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:10,763 INFO [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:10,763 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=6842bfd482b36498249bfd93acbcd189, ASSIGN in 344 msec 2023-07-11 14:21:10,763 DEBUG [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d/f2 2023-07-11 14:21:10,764 DEBUG [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d/f2 2023-07-11 14:21:10,764 INFO [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7c7523fe03e3121494af249521c3c9d columnFamilyName f2 2023-07-11 14:21:10,764 INFO [StoreOpener-e7c7523fe03e3121494af249521c3c9d-1] regionserver.HStore(310): Store=e7c7523fe03e3121494af249521c3c9d/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:10,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:10,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:10,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:21:10,769 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:10,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened e7c7523fe03e3121494af249521c3c9d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75026957, jitterRate=0.11798878014087677}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:21:10,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for e7c7523fe03e3121494af249521c3c9d: 2023-07-11 14:21:10,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d., pid=33, masterSystemTime=1689085270731 2023-07-11 14:21:10,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:10,772 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:10,772 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:10,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ceb39529e4f4e8405895377233923b91, NAME => 'testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91.', STARTKEY => '', ENDKEY => 'a'} 2023-07-11 14:21:10,772 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=e7c7523fe03e3121494af249521c3c9d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:10,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:10,772 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689085270772"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085270772"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085270772"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085270772"}]},"ts":"1689085270772"} 2023-07-11 14:21:10,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-11 14:21:10,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:10,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:10,773 INFO [StoreOpener-ceb39529e4f4e8405895377233923b91-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:10,774 DEBUG [StoreOpener-ceb39529e4f4e8405895377233923b91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91/f1 2023-07-11 14:21:10,774 DEBUG [StoreOpener-ceb39529e4f4e8405895377233923b91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91/f1 2023-07-11 14:21:10,775 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=30 2023-07-11 14:21:10,775 INFO [StoreOpener-ceb39529e4f4e8405895377233923b91-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ceb39529e4f4e8405895377233923b91 columnFamilyName f1 2023-07-11 14:21:10,775 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=30, state=SUCCESS; OpenRegionProcedure e7c7523fe03e3121494af249521c3c9d, server=jenkins-hbase9.apache.org,46501,1689085264192 in 192 msec 2023-07-11 14:21:10,775 INFO [StoreOpener-ceb39529e4f4e8405895377233923b91-1] regionserver.HStore(310): Store=ceb39529e4f4e8405895377233923b91/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:10,775 INFO [StoreOpener-ceb39529e4f4e8405895377233923b91-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:10,776 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=e7c7523fe03e3121494af249521c3c9d, ASSIGN in 357 msec 2023-07-11 14:21:10,776 DEBUG [StoreOpener-ceb39529e4f4e8405895377233923b91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91/f2 2023-07-11 14:21:10,776 DEBUG [StoreOpener-ceb39529e4f4e8405895377233923b91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91/f2 2023-07-11 14:21:10,777 INFO [StoreOpener-ceb39529e4f4e8405895377233923b91-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ceb39529e4f4e8405895377233923b91 columnFamilyName f2 2023-07-11 14:21:10,777 INFO [StoreOpener-ceb39529e4f4e8405895377233923b91-1] regionserver.HStore(310): Store=ceb39529e4f4e8405895377233923b91/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-11 14:21:10,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:10,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:10,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-11 14:21:10,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:10,781 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened ceb39529e4f4e8405895377233923b91; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68248486, jitterRate=0.016981691122055054}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-11 14:21:10,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for ceb39529e4f4e8405895377233923b91: 2023-07-11 14:21:10,782 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91., pid=35, masterSystemTime=1689085270731 2023-07-11 14:21:10,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:10,783 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:10,783 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=ceb39529e4f4e8405895377233923b91, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:10,784 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689085270783"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689085270783"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689085270783"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689085270783"}]},"ts":"1689085270783"} 2023-07-11 14:21:10,786 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=35, resume processing ppid=28 2023-07-11 14:21:10,786 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, ppid=28, state=SUCCESS; OpenRegionProcedure ceb39529e4f4e8405895377233923b91, server=jenkins-hbase9.apache.org,46501,1689085264192 in 196 msec 2023-07-11 14:21:10,787 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=27 2023-07-11 14:21:10,787 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=ceb39529e4f4e8405895377233923b91, ASSIGN in 368 msec 2023-07-11 14:21:10,788 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689085270788"}]},"ts":"1689085270788"} 2023-07-11 14:21:10,789 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=ENABLED in hbase:meta 2023-07-11 14:21:10,799 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testEnableTableWithFailoverAtStep3' was successfully enabled. 2023-07-11 14:21:10,801 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 in 1.6170 sec 2023-07-11 14:21:10,976 INFO [Listener at localhost/45629] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-11 14:21:10,976 INFO [Listener at localhost/45629] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-11 14:21:10,976 DEBUG [Listener at localhost/45629] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63e5fde6 to 127.0.0.1:55314 2023-07-11 14:21:10,976 DEBUG [Listener at localhost/45629] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:21:10,976 DEBUG [Listener at localhost/45629] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-11 14:21:10,977 DEBUG [Listener at localhost/45629] util.JVMClusterUtil(257): Found active master hash=128159301, stopped=false 2023-07-11 14:21:10,977 INFO [Listener at localhost/45629] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase9.apache.org,40895,1689085264170 2023-07-11 14:21:10,979 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-11 14:21:10,979 INFO [Listener at localhost/45629] procedure2.ProcedureExecutor(629): Stopping 2023-07-11 14:21:10,979 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): regionserver:46501-0x10154b4830b0002, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-11 14:21:10,979 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:10,980 DEBUG [Listener at localhost/45629] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x11642b7f to 127.0.0.1:55314 2023-07-11 14:21:10,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:21:10,980 DEBUG [Listener at localhost/45629] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:21:10,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46501-0x10154b4830b0002, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-11 14:21:10,980 INFO [Listener at localhost/45629] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,46501,1689085264192' ***** 2023-07-11 14:21:10,980 INFO [Listener at localhost/45629] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-11 14:21:10,980 INFO [RS:0;jenkins-hbase9:46501] regionserver.HeapMemoryManager(220): Stopping 2023-07-11 14:21:10,981 INFO [RS:0;jenkins-hbase9:46501] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-11 14:21:10,981 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-11 14:21:10,981 INFO [RS:0;jenkins-hbase9:46501] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-11 14:21:10,981 INFO [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(3305): Received CLOSE for e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:10,981 INFO [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(3305): Received CLOSE for ceb39529e4f4e8405895377233923b91 2023-07-11 14:21:10,982 INFO [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(3305): Received CLOSE for 314c4626d8bf336ef8c434a723b560cf 2023-07-11 14:21:10,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing e7c7523fe03e3121494af249521c3c9d, disabling compactions & flushes 2023-07-11 14:21:10,982 INFO [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(3305): Received CLOSE for e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:10,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:10,982 INFO [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(3305): Received CLOSE for 6842bfd482b36498249bfd93acbcd189 2023-07-11 14:21:10,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:10,982 INFO [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(1144): stopping server jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:10,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. after waiting 0 ms 2023-07-11 14:21:10,982 DEBUG [RS:0;jenkins-hbase9:46501] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07f6df26 to 127.0.0.1:55314 2023-07-11 14:21:10,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:10,982 DEBUG [RS:0;jenkins-hbase9:46501] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:21:10,982 INFO [RS:0;jenkins-hbase9:46501] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-11 14:21:10,982 INFO [RS:0;jenkins-hbase9:46501] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-11 14:21:10,982 INFO [RS:0;jenkins-hbase9:46501] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-11 14:21:10,982 INFO [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-11 14:21:10,983 INFO [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(1474): Waiting on 6 regions to close 2023-07-11 14:21:10,983 DEBUG [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(1478): Online Regions={e7c7523fe03e3121494af249521c3c9d=testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d., ceb39529e4f4e8405895377233923b91=testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91., 1588230740=hbase:meta,,1.1588230740, 314c4626d8bf336ef8c434a723b560cf=hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf., e7a2de22dc894628b2b4c8217f03978b=testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b., 6842bfd482b36498249bfd93acbcd189=testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189.} 2023-07-11 14:21:10,983 DEBUG [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(1504): Waiting on 1588230740, 314c4626d8bf336ef8c434a723b560cf, 6842bfd482b36498249bfd93acbcd189, ceb39529e4f4e8405895377233923b91, e7a2de22dc894628b2b4c8217f03978b, e7c7523fe03e3121494af249521c3c9d 2023-07-11 14:21:10,983 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-11 14:21:10,983 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-11 14:21:10,983 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-11 14:21:10,983 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-11 14:21:10,983 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-11 14:21:10,983 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=15.07 KB heapSize=24.05 KB 2023-07-11 14:21:10,986 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7c7523fe03e3121494af249521c3c9d/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-07-11 14:21:10,987 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:10,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for e7c7523fe03e3121494af249521c3c9d: 2023-07-11 14:21:10,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed testEnableTableWithFailoverAtStep3,b,1689085267620.e7c7523fe03e3121494af249521c3c9d. 2023-07-11 14:21:10,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing ceb39529e4f4e8405895377233923b91, disabling compactions & flushes 2023-07-11 14:21:10,988 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:10,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:10,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. after waiting 0 ms 2023-07-11 14:21:10,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:10,994 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/ceb39529e4f4e8405895377233923b91/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-07-11 14:21:10,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:10,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for ceb39529e4f4e8405895377233923b91: 2023-07-11 14:21:10,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed testEnableTableWithFailoverAtStep3,,1689085267620.ceb39529e4f4e8405895377233923b91. 2023-07-11 14:21:10,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 314c4626d8bf336ef8c434a723b560cf, disabling compactions & flushes 2023-07-11 14:21:10,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf. 2023-07-11 14:21:10,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf. 2023-07-11 14:21:10,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf. after waiting 0 ms 2023-07-11 14:21:10,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf. 2023-07-11 14:21:10,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 314c4626d8bf336ef8c434a723b560cf 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-11 14:21:11,048 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-11 14:21:11,183 DEBUG [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(1504): Waiting on 1588230740, 314c4626d8bf336ef8c434a723b560cf, 6842bfd482b36498249bfd93acbcd189, e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:11,238 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-11 14:21:11,238 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-11 14:21:11,239 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testEnableTableWithFailoverAtStep3' 2023-07-11 14:21:11,244 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-11 14:21:11,244 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-11 14:21:11,384 DEBUG [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(1504): Waiting on 1588230740, 314c4626d8bf336ef8c434a723b560cf, 6842bfd482b36498249bfd93acbcd189, e7a2de22dc894628b2b4c8217f03978b 2023-07-11 14:21:11,395 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=14.53 KB at sequenceid=41 (bloomFilter=false), to=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/.tmp/info/078061283f8c4ce88c18805728e2c268 2023-07-11 14:21:11,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/namespace/314c4626d8bf336ef8c434a723b560cf/.tmp/info/12ec88d3d3384a2aacb5de6619c28cb3 2023-07-11 14:21:11,410 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/namespace/314c4626d8bf336ef8c434a723b560cf/.tmp/info/12ec88d3d3384a2aacb5de6619c28cb3 as hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/namespace/314c4626d8bf336ef8c434a723b560cf/info/12ec88d3d3384a2aacb5de6619c28cb3 2023-07-11 14:21:11,414 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/namespace/314c4626d8bf336ef8c434a723b560cf/info/12ec88d3d3384a2aacb5de6619c28cb3, entries=2, sequenceid=6, filesize=4.8 K 2023-07-11 14:21:11,415 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 314c4626d8bf336ef8c434a723b560cf in 420ms, sequenceid=6, compaction requested=false 2023-07-11 14:21:11,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/namespace/314c4626d8bf336ef8c434a723b560cf/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-11 14:21:11,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf. 2023-07-11 14:21:11,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 314c4626d8bf336ef8c434a723b560cf: 2023-07-11 14:21:11,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689085266367.314c4626d8bf336ef8c434a723b560cf. 2023-07-11 14:21:11,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing e7a2de22dc894628b2b4c8217f03978b, disabling compactions & flushes 2023-07-11 14:21:11,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:11,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:11,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. after waiting 0 ms 2023-07-11 14:21:11,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:11,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/e7a2de22dc894628b2b4c8217f03978b/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-07-11 14:21:11,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:11,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for e7a2de22dc894628b2b4c8217f03978b: 2023-07-11 14:21:11,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed testEnableTableWithFailoverAtStep3,a,1689085267620.e7a2de22dc894628b2b4c8217f03978b. 2023-07-11 14:21:11,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 6842bfd482b36498249bfd93acbcd189, disabling compactions & flushes 2023-07-11 14:21:11,424 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:11,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:11,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. after waiting 0 ms 2023-07-11 14:21:11,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:11,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/default/testEnableTableWithFailoverAtStep3/6842bfd482b36498249bfd93acbcd189/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-07-11 14:21:11,427 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:11,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 6842bfd482b36498249bfd93acbcd189: 2023-07-11 14:21:11,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed testEnableTableWithFailoverAtStep3,c,1689085267620.6842bfd482b36498249bfd93acbcd189. 2023-07-11 14:21:11,584 DEBUG [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-11 14:21:11,784 DEBUG [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-11 14:21:11,807 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=556 B at sequenceid=41 (bloomFilter=false), to=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/.tmp/table/d6c42c2ec00c4c0ab8b01fd0cab034e4 2023-07-11 14:21:11,814 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/.tmp/info/078061283f8c4ce88c18805728e2c268 as hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/info/078061283f8c4ce88c18805728e2c268 2023-07-11 14:21:11,818 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/info/078061283f8c4ce88c18805728e2c268, entries=70, sequenceid=41, filesize=14.7 K 2023-07-11 14:21:11,819 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/.tmp/table/d6c42c2ec00c4c0ab8b01fd0cab034e4 as hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/table/d6c42c2ec00c4c0ab8b01fd0cab034e4 2023-07-11 14:21:11,824 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/table/d6c42c2ec00c4c0ab8b01fd0cab034e4, entries=5, sequenceid=41, filesize=4.9 K 2023-07-11 14:21:11,825 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~15.07 KB/15431, heapSize ~23.77 KB/24336, currentSize=0 B/0 for 1588230740 in 842ms, sequenceid=41, compaction requested=false 2023-07-11 14:21:11,833 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/data/hbase/meta/1588230740/recovered.edits/44.seqid, newMaxSeqId=44, maxSeqId=1 2023-07-11 14:21:11,833 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-11 14:21:11,834 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-11 14:21:11,834 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-11 14:21:11,834 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-11 14:21:11,984 INFO [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,46501,1689085264192; all regions closed. 2023-07-11 14:21:11,988 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/WALs/jenkins-hbase9.apache.org,46501,1689085264192/jenkins-hbase9.apache.org%2C46501%2C1689085264192.meta.1689085266315.meta not finished, retry = 0 2023-07-11 14:21:12,091 DEBUG [RS:0;jenkins-hbase9:46501] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/oldWALs 2023-07-11 14:21:12,091 INFO [RS:0;jenkins-hbase9:46501] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C46501%2C1689085264192.meta:.meta(num 1689085266315) 2023-07-11 14:21:12,095 DEBUG [RS:0;jenkins-hbase9:46501] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/oldWALs 2023-07-11 14:21:12,095 INFO [RS:0;jenkins-hbase9:46501] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C46501%2C1689085264192:(num 1689085265370) 2023-07-11 14:21:12,095 DEBUG [RS:0;jenkins-hbase9:46501] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:21:12,096 INFO [RS:0;jenkins-hbase9:46501] regionserver.LeaseManager(133): Closed leases 2023-07-11 14:21:12,096 INFO [RS:0;jenkins-hbase9:46501] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-11 14:21:12,096 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 14:21:12,098 INFO [RS:0;jenkins-hbase9:46501] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:46501 2023-07-11 14:21:12,099 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): regionserver:46501-0x10154b4830b0002, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,46501,1689085264192 2023-07-11 14:21:12,100 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 14:21:12,100 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): regionserver:46501-0x10154b4830b0002, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-11 14:21:12,102 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,46501,1689085264192] 2023-07-11 14:21:12,102 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,46501,1689085264192; numProcessing=1 2023-07-11 14:21:12,104 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,46501,1689085264192 already deleted, retry=false 2023-07-11 14:21:12,104 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,46501,1689085264192 expired; onlineServers=0 2023-07-11 14:21:12,104 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase9.apache.org,40895,1689085264170' ***** 2023-07-11 14:21:12,104 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-11 14:21:12,104 DEBUG [M:1;jenkins-hbase9:40895] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39c0a69, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-07-11 14:21:12,104 INFO [M:1;jenkins-hbase9:40895] regionserver.HRegionServer(1144): stopping server jenkins-hbase9.apache.org,40895,1689085264170 2023-07-11 14:21:12,104 INFO [M:1;jenkins-hbase9:40895] regionserver.HRegionServer(1170): stopping server jenkins-hbase9.apache.org,40895,1689085264170; all regions closed. 2023-07-11 14:21:12,104 DEBUG [M:1;jenkins-hbase9:40895] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-11 14:21:12,104 DEBUG [M:1;jenkins-hbase9:40895] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-11 14:21:12,104 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-11 14:21:12,104 DEBUG [M:1;jenkins-hbase9:40895] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-11 14:21:12,104 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085270388] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1689085270388,5,FailOnTimeoutGroup] 2023-07-11 14:21:12,104 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085270388] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1689085270388,5,FailOnTimeoutGroup] 2023-07-11 14:21:12,104 INFO [M:1;jenkins-hbase9:40895] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-11 14:21:12,106 INFO [M:1;jenkins-hbase9:40895] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-11 14:21:12,106 INFO [M:1;jenkins-hbase9:40895] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-07-11 14:21:12,106 DEBUG [M:1;jenkins-hbase9:40895] master.HMaster(1512): Stopping service threads 2023-07-11 14:21:12,106 INFO [M:1;jenkins-hbase9:40895] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-11 14:21:12,106 ERROR [M:1;jenkins-hbase9:40895] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-11 14:21:12,107 INFO [M:1;jenkins-hbase9:40895] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-11 14:21:12,107 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-11 14:21:12,108 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-11 14:21:12,108 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-11 14:21:12,109 DEBUG [M:1;jenkins-hbase9:40895] zookeeper.ZKUtil(398): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-11 14:21:12,109 WARN [M:1;jenkins-hbase9:40895] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-11 14:21:12,109 INFO [M:1;jenkins-hbase9:40895] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-11 14:21:12,109 INFO [M:1;jenkins-hbase9:40895] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-11 14:21:12,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-11 14:21:12,113 DEBUG [M:1;jenkins-hbase9:40895] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-11 14:21:12,113 INFO [M:1;jenkins-hbase9:40895] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:12,113 DEBUG [M:1;jenkins-hbase9:40895] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:12,113 DEBUG [M:1;jenkins-hbase9:40895] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-11 14:21:12,113 DEBUG [M:1;jenkins-hbase9:40895] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:12,113 INFO [M:1;jenkins-hbase9:40895] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=24.96 KB heapSize=30.73 KB 2023-07-11 14:21:12,130 INFO [M:1;jenkins-hbase9:40895] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.96 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b42c6c4b6d1f467c91eee90ec80b690c 2023-07-11 14:21:12,145 DEBUG [M:1;jenkins-hbase9:40895] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b42c6c4b6d1f467c91eee90ec80b690c as hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b42c6c4b6d1f467c91eee90ec80b690c 2023-07-11 14:21:12,151 INFO [M:1;jenkins-hbase9:40895] regionserver.HStore(1080): Added hdfs://localhost:43599/user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b42c6c4b6d1f467c91eee90ec80b690c, entries=9, sequenceid=254, filesize=5.4 K 2023-07-11 14:21:12,152 INFO [M:1;jenkins-hbase9:40895] regionserver.HRegion(2948): Finished flush of dataSize ~24.96 KB/25554, heapSize ~30.71 KB/31448, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 39ms, sequenceid=254, compaction requested=false 2023-07-11 14:21:12,160 INFO [M:1;jenkins-hbase9:40895] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-11 14:21:12,160 DEBUG [M:1;jenkins-hbase9:40895] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-11 14:21:12,163 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/cfebee27-4131-1d41-3342-f6323ff5d1e5/MasterData/WALs/jenkins-hbase9.apache.org,40895,1689085264170/jenkins-hbase9.apache.org%2C40895%2C1689085264170.1689085270283 not finished, retry = 0 2023-07-11 14:21:12,202 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): regionserver:46501-0x10154b4830b0002, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:21:12,202 INFO [RS:0;jenkins-hbase9:46501] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,46501,1689085264192; zookeeper connection closed. 2023-07-11 14:21:12,202 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): regionserver:46501-0x10154b4830b0002, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:21:12,203 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4290d475] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4290d475 2023-07-11 14:21:12,203 INFO [Listener at localhost/45629] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 1 regionserver(s) complete 2023-07-11 14:21:12,265 INFO [M:1;jenkins-hbase9:40895] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-11 14:21:12,265 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-11 14:21:12,265 INFO [M:1;jenkins-hbase9:40895] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:40895 2023-07-11 14:21:12,268 DEBUG [M:1;jenkins-hbase9:40895] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,40895,1689085264170 already deleted, retry=false 2023-07-11 14:21:12,370 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:21:12,370 INFO [M:1;jenkins-hbase9:40895] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase9.apache.org,40895,1689085264170; zookeeper connection closed. 2023-07-11 14:21:12,370 DEBUG [Listener at localhost/45629-EventThread] zookeeper.ZKWatcher(600): master:40895-0x10154b4830b0001, quorum=127.0.0.1:55314, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-11 14:21:12,371 WARN [Listener at localhost/45629] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-11 14:21:12,375 INFO [Listener at localhost/45629] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-11 14:21:12,479 WARN [BP-166859448-172.31.2.10-1689085263316 heartbeating to localhost/127.0.0.1:43599] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-11 14:21:12,479 WARN [BP-166859448-172.31.2.10-1689085263316 heartbeating to localhost/127.0.0.1:43599] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-166859448-172.31.2.10-1689085263316 (Datanode Uuid 68c167b7-9d05-40f2-8dc8-414baa8e4181) service to localhost/127.0.0.1:43599 2023-07-11 14:21:12,480 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/cluster_b3df6d01-63d8-53bf-20c3-0b45a541bf8e/dfs/data/data1/current/BP-166859448-172.31.2.10-1689085263316] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-11 14:21:12,480 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fd85d19d-5967-0a04-258f-532eb9a71934/cluster_b3df6d01-63d8-53bf-20c3-0b45a541bf8e/dfs/data/data2/current/BP-166859448-172.31.2.10-1689085263316] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-11 14:21:12,492 INFO [Listener at localhost/45629] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-11 14:21:12,602 INFO [Listener at localhost/45629] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-11 14:21:12,616 INFO [Listener at localhost/45629] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-07-11 14:21:12,631 INFO [Listener at localhost/45629] hbase.ResourceChecker(175): after: master.procedure.TestMasterFailoverWithProcedures#testEnableTableWithFailover Thread=149 (was 123) Potentially hanging thread: nioEventLoopGroup-11-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (200715596) connection to localhost/127.0.0.1:43599 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-10-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-17-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (200715596) connection to localhost/127.0.0.1:43599 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-17-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (200715596) connection to localhost/127.0.0.1:43599 from jenkins.hfs.4 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-10-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-17-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/45629 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:43599 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.4@localhost:43599 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=504 (was 482) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=186 (was 193), ProcessCount=171 (was 171), AvailableMemoryMB=8096 (was 7294) - AvailableMemoryMB LEAK? -